LARTC - Oct 2005 - Inaccurate results without natting the traffic.

Hi folks.


For quite a while, I tried to use linux (cbq and htb) to control
bandwidth in order to replace a cisco equipment used to play this role
on our network.

After trying here and there, with a bunch of different distros,
different versions, different compilations, ... the closest I could
get was as follows:

(using htb.init and cbq.init)

Desired BW ==> Achieved BW

128k ==> 404.78 KB/s
64k ==> 337.9 KB/s
16k ==> 68.86 KB/s
8k ==> 31.12 KB/s
1k ==> 3.77 KB/s

No matter what I''d do, the results were always the same, i.e, control
out of control.

I tried to get some help from a few mailing lists, including this one.

Anyway, one thing I noticed was that all the documentation regarding
the matter, and most of the messages on the mailing lists would
mention firewalling the box along with the traffic control.

I asked if the firewall was a mandatory piece on this puzzle, and got
no as answer. Without giving up, I stayed playing (actually it has
been a fight ....) in the lab installing and configuring a linux box
to replace our cisco bbsm. Very frustrating task, since I was like
running after the tail.... Iptables were configured in a bunch of
ways, .... no result.

Until the day, for no particular reason, I decided to masquerade the
traffic. Oh men, I got so desperated and mad when I saw the box
controlling the traffic as it should.....

The reason for such a reaction has a simple answer: I cannot
masquerade the traffic..... :o(((((

I managed to find a company that sells a solution (built on top of
conectiva) that does control the traffic without using masquerade. I
even played with the solution and saw that it does not nat the traffic
...... but the guys did not disclose to me what was the trick... and
since we buried a lot of money on cisco''s solution, no one here is
willing to spend money to replace it. This decision is out of my
hands.

Now I come to you guys again, with the question: How to make cbq or
htb work without masquerading the traffic?


Any help is very appreciated.


Cheers,


AL

Alvaro Motta wrote:
> Now I come to you guys again, with the question: How to make cbq or
> htb work without masquerading the traffic?
That really isn''t supposed to have anything to do with it.... Can you
post your configuration files/scripts and a quick ASCII map showing
where your Linux machine is placed?

-Corey

Hi Corey,

First, let me thank you for your message. Sorry for not responding before.

As per your request, here is a sketch of my test bed environment.


         ____            ____            ____
        |       |          |       |           |       |
        | A    |----------| B    |----------| C    |
        |____|          |____|           |____|



Machine A
IPA - 192.168.0.20/24
DefGw - 192.168.0.1

Machine B
IPB (eth1) - 192.168.0.19/24
IPB (eth0) - 10.4.10.201/16
DefGw - 192.168.0.1

Machine C
IPC - 10.4.10.202/16
DefGw - 57.4.10.202


On machine B:

2.6.11-1.1369_FC4 (Fedora Core 4)
iproute-2.6.11-1
cbq.init v0.7.3

following scripts:
cbq-0002.eth0
DEVICE=eth0, 100Mbit,10Mbit
RATE=100Mbit
WEIGHT=10Mbit

cbq-0003.eth1
DEVICE=eth1, 100Mbit,10Mbit
RATE=100Mbit
WEIGHT=10Mbit

cbq-0004.dwn
DEVICE=eth0
RATE=64Kbit
WEIGHT=6kbit
RULE=0/0,0/0

cbq-0005.upl
DEVICE=eth1
RATE=64kbit
WEIGHT=6kbit
RULE=0/0,0/0


For testing purposes, we have apache installed on machines A and
C.
>From C I do a wget 192.168.0.20/1M and get those funk results stated
on my previous message.
>From A I do a wget 10.4.10.202/1M and the achieved results are very
close to the desired bandwidth.

So, from C''s perspective, the upload rate is fine, but the download
rate is way out of what we want.

The funniest thing is that if I swap the interfaces on the .dwn and
.upl files, the results are the same.

If, on B I issue iptables -t nat -A POSTROUTING -s 0/0 -d 0/0 -j
MASQUERADE, then the results of download and upload are

pretty much the same.


Whatever I do with the iptables command, the only way to have the
desired transfer rate is if the target and destination machines are
included in the nat (that''s why I use 0/0).


Output of cbq.init compile:

/sbin/tc qdisc del dev eth0 root
/sbin/tc qdisc add dev eth0 root handle 1 cbq bandwidth 100Mbit avpkt
1000 cell 8
/sbin/tc class change dev eth0 root cbq weight 10Mbit allot 1514

/sbin/tc qdisc del dev eth1 root
/sbin/tc qdisc add dev eth1 root handle 1 cbq bandwidth 100Mbit avpkt
1000 cell 8
/sbin/tc class change dev eth1 root cbq weight 10Mbit allot 1514

/sbin/tc class add dev eth0 parent 1: classid 1:2 cbq bandwidth
100Mbit rate 100Mbit weight 10Mbit prio 5 allot 1514 cell 8 maxburst
20 avpkt 1000 bounded
/sbin/tc qdisc add dev eth0 parent 1:2 handle 2 tbf rate 100Mbit
buffer 10Kb/8 limit 15Kb mtu 1500

/sbin/tc class add dev eth1 parent 1: classid 1:3 cbq bandwidth
100Mbit rate 100Mbit weight 10Mbit prio 5 allot 1514 cell 8 maxburst
20 avpkt 1000 bounded
/sbin/tc qdisc add dev eth1 parent 1:3 handle 3 tbf rate 100Mbit
buffer 10Kb/8 limit 15Kb mtu 1500

/sbin/tc class add dev eth0 parent 1: classid 1:4 cbq bandwidth
100Mbit rate 64Kbit weight 6Kbit prio 5 allot 1514 cell 8 maxburst 20
avpkt 1000 bounded
/sbin/tc qdisc add dev eth0 parent 1:4 handle 4 tbf rate 64Kbit buffer
10Kb/8 limit 15Kb mtu 1500
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 200 u32 match
ip src 0/0 match ip dst 0/0 classid 1:4

/sbin/tc class add dev eth1 parent 1: classid 1:5 cbq bandwidth
100Mbit rate 64Kbit weight 6Kbit prio 5 allot 1514 cell 8 maxburst 20
avpkt 1000 bounded
/sbin/tc qdisc add dev eth1 parent 1:5 handle 5 tbf rate 64Kbit buffer
10Kb/8 limit 15Kb mtu 1500
/sbin/tc filter add dev eth1 parent 1:0 protocol ip prio 200 u32 match
ip src 0/0 match ip dst 0/0 classid 1:5



Output of cbq.init list

### eth0: queueing disciplines

qdisc cbq 1: rate 100000Kbit (bounded,isolated) prio no-transmit
qdisc tbf 2: parent 1:2 rate 100000Kbit burst 10Kb lat 400us
qdisc tbf 4: parent 1:4 rate 64000bit burst 10Kb lat 625.0ms

### eth0: traffic classes

class cbq 1: root rate 100000Kbit (bounded,isolated) prio no-transmit
class cbq 1:2 parent 1: leaf 2: rate 100000Kbit (bounded) prio 5
class cbq 1:4 parent 1: leaf 4: rate 64000bit (bounded) prio 5
class tbf 2:1 parent 2:
class tbf 4:1 parent 4:

### eth0: filtering rules

filter parent 1: protocol ip pref 200 u32
filter parent 1: protocol ip pref 200 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 200 u32 fh 800::800 order 2048 key
ht 800 bkt 0 flowid 1:4
  match 00000000/00000000 at 12
  match 00000000/00000000 at 16

### eth1: queueing disciplines

qdisc cbq 1: rate 100000Kbit (bounded,isolated) prio no-transmit
qdisc tbf 3: parent 1:3 rate 100000Kbit burst 10Kb lat 400us
qdisc tbf 5: parent 1:5 rate 64000bit burst 10Kb lat 625.0ms

### eth1: traffic classes

class cbq 1: root rate 100000Kbit (bounded,isolated) prio no-transmit
class cbq 1:3 parent 1: leaf 3: rate 100000Kbit (bounded) prio 5
class cbq 1:5 parent 1: leaf 5: rate 64000bit (bounded) prio 5
class tbf 3:1 parent 3:
class tbf 5:1 parent 5:

### eth1: filtering rules

filter parent 1: protocol ip pref 200 u32
filter parent 1: protocol ip pref 200 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 200 u32 fh 800::800 order 2048 key
ht 800 bkt 0 flowid 1:5
  match 00000000/00000000 at 12
  match 00000000/00000000 at 16



Output of cbq.init stats

### eth0: queueing disciplines

qdisc cbq 1: rate 100000Kbit (bounded,isolated) prio no-transmit
 Sent 84 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 81 undertime 0
qdisc tbf 2: parent 1:2 rate 100000Kbit burst 10Kb lat 400us
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc tbf 4: parent 1:4 rate 64000bit burst 10Kb lat 625.0ms
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

### eth0: traffic classes

class cbq 1: root rate 100000Kbit (bounded,isolated) prio no-transmit
 Sent 84 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 81 undertime 0
class cbq 1:2 parent 1: leaf 2: rate 100000Kbit (bounded) prio 5
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 81 undertime 0
class cbq 1:4 parent 1: leaf 4: rate 64000bit (bounded) prio 5
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 3.63072e+06 undertime 0
class tbf 2:1 parent 2:

class tbf 4:1 parent 4:


### eth0: filtering rules

filter parent 1: protocol ip pref 200 u32
filter parent 1: protocol ip pref 200 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 200 u32 fh 800::800 order 2048 key
ht 800 bkt 0 flowid 1:4  (rule hit 0 success 0)
  match 00000000/00000000 at 12 (success 0 )
  match 00000000/00000000 at 16 (success 0 )

### eth1: queueing disciplines

qdisc cbq 1: rate 100000Kbit (bounded,isolated) prio no-transmit
 Sent 2352 bytes 56 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 81 undertime 0
qdisc tbf 3: parent 1:3 rate 100000Kbit burst 10Kb lat 400us
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc tbf 5: parent 1:5 rate 64000bit burst 10Kb lat 625.0ms
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

### eth1: traffic classes

class cbq 1: root rate 100000Kbit (bounded,isolated) prio no-transmit
 Sent 2352 bytes 56 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 81 undertime 0
class cbq 1:3 parent 1: leaf 3: rate 100000Kbit (bounded) prio 5
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 81 undertime 0
class cbq 1:5 parent 1: leaf 5: rate 64000bit (bounded) prio 5
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
  borrowed 0 overactions 0 avgidle 3.63072e+06 undertime 0
class tbf 3:1 parent 3:

class tbf 5:1 parent 5:


### eth1: filtering rules

filter parent 1: protocol ip pref 200 u32
filter parent 1: protocol ip pref 200 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 200 u32 fh 800::800 order 2048 key
ht 800 bkt 0 flowid 1:5  (rule hit 0 success 0)
  match 00000000/00000000 at 12 (success 0 )
  match 00000000/00000000 at 16 (success 0 )




I hope this is enough information for you to try to help me.


Thanks for your time and consideration.


best regards,


AL

On 10/11/05, Corey Hickey <bugfood-ml@fatooh.org>
wrote:
> Alvaro Motta wrote:
> > Now I come to you guys again, with the question: How to make cbq or
> > htb work without masquerading the traffic?
>
> That really isn''t supposed to have anything to do with it.... Can
you
> post your configuration files/scripts and a quick ASCII map showing
> where your Linux machine is placed?
>
> -Corey
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>

Alvaro Motta wrote:
> I hope this is enough information for you to try to help me.
> 
> 
> Thanks for your time and consideration.
Well, it looks like you''ve posted enough information now.
Unfortunately,
I''m not having any bright ideas. You have me stumped. Hopefully someone
else on this list will take interest and be able to help you.

Sorry,
Corey