LARTC - Jul 2005 - Problems setting up nested qdisc, feedback to LARTC HOWTO

Hi all,

based on the information in the "Linux Advanced Routing & Traffic
Control HOWTO", I was trying to set up traffic shaping on my firewall.

While I found the HOWTO very useful, in the process I ran into some
problems that I did not forsee: According to the HOWTO it seems that
it should have worked, even after spending some time going through the
sections looking for answers, the problem was not obvious to me.

So I would appreciate if you could tell me where my mistake was and
also maybe add a bit of information to the HOWTO to help others fall
into the same traps that I fell into. :-)

Here is what I wanted my ideal solution to look like:

A strong priority of traffic, where parts of the upstream should be
guaranteed rate for some traffic, the rest should be given to normal
traffic and any "leftovers" to BULK traffic, which is allowed to
starve for a while. Also, connection handshake and such very short,
time critical things should get absolute priority over everything
else.

So this is what I ideally wanted to set up:

 1: PRIO QDISC (4 Bands), DEFAULT: ALL TO BAND 3 (2 in priomap)

 1:1 -> SFQ, handle 10:
             for priority communication (connection handshake & co)

 1:2 -> HTB, handle 20:
             limited to Xk for different kinds of guaranteed rates that
             can "borrow" from each other, but never more than the
             maximum -- so it cannot saturate the link fully.

        20:1 -> SFQ, handle 100:
        20:2 -> SFQ, handle 200:
        20:3 -> SFQ, handle 300:
        20:4 -> SFQ, handle 400:
        [...]

 1:3 -> PRIO QDISC (default), handle 30:
        for all "normal"/unclassified traffic, TOS splitting only
        30:1 (BAND 1)
        30:2 (BAND 2)
        30:3 (BAND 3)

 1:4 -> PFIFO, handle 40:
        "starvation bitbucket"
        gets what is left, can starve at times

The setup was apparently successful, tc does not complain and displays
the structure: without any CLASSIFY targets, all traffic goes to 1:3
and is split in the three bands properly. Looks good.

I can also add CLASSIFY for 1:1 and 1:4, which seem to fall into the
SFQ/PFIFO buckets underneath, both look good.

The problem starts with the HTB embedded in 1:2 as 20: -- I can never
CLASSIFY traffic for it properly, the only way to get ANY traffic into
it is to CLASSIFY for 1:2, but that puts all into its default bucket,
which defeats its purpose.

Neither 20:1 nor 100:0 would put traffic into its first bucket.

Such classification is simply ignored as far as I can tell.

This is the problem that I ultimately did not find a way to solve, the
"indirect" approach of using the MARK target and tc filters also did
not work -- it shows the exact same result.

I currently run another approach to this, which I am not quite as
happy with, but which works for now -- but would still like to know:

 * WHY was the 20:1 or 100:0 CLASSIFY not successful? Nothing in the
   documentation seemed to indicate that it should fail.

 * HOW could it have been made to work?

 * WHAT kind of information was I lacking?

or, in short: WHAT did I do wrong?

I''d be grateful to find an answer and think it might help to then find
a way to add that answer into the LARTC HOWTO.

Regards,
Georg

-- 
Georg C. F. Greve                                 <greve@fsfeurope.org>
Free Software Foundation Europe	                 (http://fsfeurope.org)
Join the Fellowship and protect your freedom!     (http://www.fsfe.org)


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Have you tried not using classify but instead using tc filters?  Maybe
this is a limitation with iptables classify.  Try using your classify
to put things into 20:  and then use tc filters attached to 20: to
split into the htb subclasses?

I never used classify much and have always used tc filter instead, and
i have done a setup similar to this with success.

- Jody

On 7/15/05, Georg C. F. Greve <greve@fsfeurope.org>
wrote:
> Hi all,
> 
> based on the information in the "Linux Advanced Routing & Traffic
> Control HOWTO", I was trying to set up traffic shaping on my firewall.
> 
> While I found the HOWTO very useful, in the process I ran into some
> problems that I did not forsee: According to the HOWTO it seems that
> it should have worked, even after spending some time going through the
> sections looking for answers, the problem was not obvious to me.
> 
> So I would appreciate if you could tell me where my mistake was and
> also maybe add a bit of information to the HOWTO to help others fall
> into the same traps that I fell into. :-)
> 
> Here is what I wanted my ideal solution to look like:
> 
> A strong priority of traffic, where parts of the upstream should be
> guaranteed rate for some traffic, the rest should be given to normal
> traffic and any "leftovers" to BULK traffic, which is allowed to
> starve for a while. Also, connection handshake and such very short,
> time critical things should get absolute priority over everything
> else.
> 
> So this is what I ideally wanted to set up:
> 
>  1: PRIO QDISC (4 Bands), DEFAULT: ALL TO BAND 3 (2 in priomap)
> 
>  1:1 -> SFQ, handle 10:
>              for priority communication (connection handshake & co)
> 
>  1:2 -> HTB, handle 20:
>              limited to Xk for different kinds of guaranteed rates that
>              can "borrow" from each other, but never more than
the
>              maximum -- so it cannot saturate the link fully.
> 
>         20:1 -> SFQ, handle 100:
>         20:2 -> SFQ, handle 200:
>         20:3 -> SFQ, handle 300:
>         20:4 -> SFQ, handle 400:
>         [...]
> 
>  1:3 -> PRIO QDISC (default), handle 30:
>         for all "normal"/unclassified traffic, TOS splitting only
>         30:1 (BAND 1)
>         30:2 (BAND 2)
>         30:3 (BAND 3)
> 
>  1:4 -> PFIFO, handle 40:
>         "starvation bitbucket"
>         gets what is left, can starve at times
> 
> The setup was apparently successful, tc does not complain and displays
> the structure: without any CLASSIFY targets, all traffic goes to 1:3
> and is split in the three bands properly. Looks good.
> 
> I can also add CLASSIFY for 1:1 and 1:4, which seem to fall into the
> SFQ/PFIFO buckets underneath, both look good.
> 
> The problem starts with the HTB embedded in 1:2 as 20: -- I can never
> CLASSIFY traffic for it properly, the only way to get ANY traffic into
> it is to CLASSIFY for 1:2, but that puts all into its default bucket,
> which defeats its purpose.
> 
> Neither 20:1 nor 100:0 would put traffic into its first bucket.
> 
> Such classification is simply ignored as far as I can tell.
> 
> This is the problem that I ultimately did not find a way to solve, the
> "indirect" approach of using the MARK target and tc filters also
did
> not work -- it shows the exact same result.
> 
> I currently run another approach to this, which I am not quite as
> happy with, but which works for now -- but would still like to know:
> 
>  * WHY was the 20:1 or 100:0 CLASSIFY not successful? Nothing in the
>    documentation seemed to indicate that it should fail.
> 
>  * HOW could it have been made to work?
> 
>  * WHAT kind of information was I lacking?
> 
> or, in short: WHAT did I do wrong?
> 
> I''d be grateful to find an answer and think it might help to then
find
> a way to add that answer into the LARTC HOWTO.
> 
> Regards,
> Georg
> 
> --
> Georg C. F. Greve                                
<greve@fsfeurope.org>
> Free Software Foundation Europe                  (http://fsfeurope.org)
> Join the Fellowship and protect your freedom!     (http://www.fsfe.org)
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 
> 
> 
>

|| On Fri, 15 Jul 2005 14:46:24 -0400
 || Jody Shumaker <jody.shumaker@gmail.com> wrote: 

 js> Have you tried not using classify but instead using tc filters?

Well, the tc filters are much more limited than iptables, they could
not replicate what I am doing with the iptables. That is why I tried
using the -j MARK iptables target to set a tc filter based on that.


 js> Maybe this is a limitation with iptables classify.  Try using
 js> your classify to put things into 20: and then use tc filters
 js> attached to 20: to split into the htb subclasses?

Interesting idea.

It is comforting that you have done something similar with success, so
I guess a combination of -j MARK and -j CLASSIFY targets might be able
to do that job, I will have to try this.

But having to employ such a mix seems like a cludge, shouldn''t this
work properly with CLASSIFY, as well? Nothing in the documentation
says that it shouldn''t -- and the docs are missing sufficiently
complex examples to get an idea of how others solved that problem.

It seems some problem exists, it is just not clear to me yet whether
this is a bug in the documentation or the software.

Regards,
Georg

-- 
Georg C. F. Greve                                 <greve@fsfeurope.org>
Free Software Foundation Europe	                 (http://fsfeurope.org)
Join the Fellowship and protect your freedom!     (http://www.fsfe.org)


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

|| On Fri, 15 Jul 2005 22:58:09 +0200
 || "Georg C. F. Greve" <greve@fsfeurope.org> wrote: 

 gg> It is comforting that you have done something similar with
 gg> success, so I guess a combination of -j MARK and -j CLASSIFY
 gg> targets might be able to do that job, I will have to try this.

UPDATE: This indeed appears to be working.

Maybe this ought to go into the HOWTO in some way.

Regards,
Georg

-- 
Georg C. F. Greve                                       <greve@gnu.org>
Free Software Foundation Europe	                 (http://fsfeurope.org)
Join the Fellowship and protect your freedom!     (http://www.fsfe.org)


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

When I read this earlier I thought of offering the same, i''m using a
mix
of -j mark and tc filters to do my routing into nexted qdiscs.  Seems 
more like i''d consider this a bug of classify that should be fixed, and
maybe a note in the howto that it is broken for now.

- Jody

Georg C. F. Greve wrote:
> || On Fri, 15 Jul 2005 22:58:09 +0200
> || "Georg C. F. Greve" <greve@fsfeurope.org> wrote: 
>
> gg> It is comforting that you have done something similar with
> gg> success, so I guess a combination of -j MARK and -j CLASSIFY
> gg> targets might be able to do that job, I will have to try this.
>
>UPDATE: This indeed appears to be working.
>
>Maybe this ought to go into the HOWTO in some way.
>
>Regards,
>Georg
>
>  
>

|| On Sat, 16 Jul 2005 01:14:48 -0400
 || Jody Shumaker <jody.shumaker@gmail.com> wrote: 

 js> When I read this earlier I thought of offering the same, i''m
 js> using a mix of -j mark and tc filters to do my routing into
 js> nexted qdiscs.  Seems more like i''d consider this a bug of
 js> classify that should be fixed, and maybe a note in the howto that
 js> it is broken for now.

I fully agree with your classification of this behaviour as a bug.

Also I agree that ideally it should be fixed.

Do the authors know this bug exists? Where would it have to be filed?


Until then, what about adding something to section 12, "Advanced
filters for (re-)classifying packets" about this? My suggestion would
be to add a subsection

 12.X Nested qdiscs: complex classification using MARK and CLASSIFY

 12.X.1 Classification problems

        In environments with nested qdiscs, the iptables CLASSIFY
        target or tc filters on root qdisc level do not work as one
        might expect: classification to assign a packet to a qdisc in
        a nested structure is ignored.

        Consider the following example:

        generate a prio qdisc, all traffic going to band 3 (which is a
        somewhat inconsistent numerical 2 in priomap, see prio qdisc
        section for more information):
          # tc qdisc add dev eth0 root handle 1: prio priomap 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2

        now add a htb qdisc at band 1 of the prio qdisc:
          # tc qdisc add dev eth0 parent 1:1 handle 10: htb default 40
          # tc class add dev eth0 parent 10: classid 10:1 htb rate 500kbit
          # tc class add dev eth0 parent 10:1 classid 10:10 htb rate 100kbit
          # tc class add dev eth0 parent 10:1 classid 10:20 htb rate 100kbit
          # tc class add dev eth0 parent 10:1 classid 10:30 htb rate 100kbit
          # tc class add dev eth0 parent 10:1 classid 10:40 htb rate 100kbit

        All unclassified traffic goes into band 3 of the prio qdisc,
        traffic classified for target 1:2 goes into band 2, as
        expected. But if traffic is classified for 10:10, 10:20, 10:30
        or 10:40, all of it goes into band 3 of the prio qdisc,
        classification is ignored!

        Yes, this looks like a bug. Fortunately, there is a
        workaround.

 12.X.2 Combining MARK and CLASSIFY

        In the above example, classifying traffic for 1:1 works as
        expected: traffic goes into the htb, all traffic ending up in
        the default bucket, 10:40. This is obviously not very useful,
        but allows us to now *attach a filter to the htb qdisc*:

          # tc filter add dev eth0 protocol ip parent 10:0 prio 1 handle 1 fw
classid 10:10

        Which puts all packets marked with handle 1 into the first
        bucket: Marking packets is done in iptables with the MARK
        target. And naturally, you can also place other filters into
        the htb qdisc.

        So for packets to be distributed in a nested hierarchy, you
        have to CLASSIFY that traffic for the root qdisc minor device
        that contains the nested structure and then add filters inside
        that nested structure which subdivide the traffic.

        The MARK target works very well for this.


Naturally, feel free to edit/improve.

Could the appropriate section author please add this? I think it would
be very helpful.

Regards,
Georg

-- 
Georg C. F. Greve                                 <greve@fsfeurope.org>
Free Software Foundation Europe	                 (http://fsfeurope.org)
Join the Fellowship and protect your freedom!     (http://www.fsfe.org)


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Georg C. F. Greve wrote:
>  || On Sat, 16 Jul 2005 01:14:48 -0400
>  || Jody Shumaker <jody.shumaker@gmail.com> wrote: 
> 
>  js> When I read this earlier I thought of offering the same,
i''m
>  js> using a mix of -j mark and tc filters to do my routing into
>  js> nexted qdiscs.  Seems more like i''d consider this a bug of
>  js> classify that should be fixed, and maybe a note in the howto that
>  js> it is broken for now.
> 
> I fully agree with your classification of this behaviour as a bug.
> 
> Also I agree that ideally it should be fixed.
I don''t know if I would call it a bug - you are nesting classfull
qdiscs
and expecting them to know about each others classes when they don''t.

You have to filter the packets from root to the leaf containing the next 
qdisc and if it is classfull you have to filter again there. This is 
never going to be doable with just iptables classify.

IIRC the example in LARTC doesn''t nest classful - I agree documenting 
your setup will be usefull - but LARTC doesn''t get updated much.

FWIW your setup just wouldn''t work for me - you are not limiting 
bandwidth for the link - you would need htb as root to do that.

In my case if I put that on my dsl connection it would go overlimits and 
I would end up with a 2 second queue in my modem.

Andy.