LARTC - Jan 2007 - egress bandwidth not limited / limited extremely inaccurately

I''m just-so-fresh to the list, so hello everyone.

I''m having a realy hard time with setting up very simple bandwidth
management. What I am trying to do is setup a 10Mbit interface to send
at only 1920kbit to most of the network (to make sure it hardly ever
tops 2mbit). I did succeed with the ingress traffic... With egress
however, I get very odd results at different speeds.
With HTB:
If i try to throttle at 0.5mbit for example, a windows box on 1mbit ADSL
line is capable of downloading at ~64k from the box i''m limiting, while
a bsd box on a 4mbit DSL line happily downloads at ~200k. Limiting to
2mbit looks like there are no results at all
With TBF:
The dowloads from the limited box are always ~5k, at least to what I''ve
observed with different configurations (even with limits at 2mbit).

I''m doing all of that directly on the Internet with public IPs.

I think I''ve read all the TLDP and LARTC to no results, so would
greatly
appreciate any help with strictly cutting the bandwidth ath ~1920kbit.
What am I missing?


The scrips I''ve tried (or tried their variants):

#tc qdisc add dev eth0 root handle 1: htb default 20
#tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit
#tc class add dev eth0 parent 1:1 classid 1:10 htb rate 9.5mbit
#tc class add dev eth0 parent 1:1 classid 1:20 htb rate 0.5mbit
#tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
#tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
#tc filter add dev eth0 parent 1: protocol ip u32 match ip dst
a.b.c.d/24 flowid 1:10

#tc qdisc add dev eth0 root tbf rate 0.5mbit burst 5kb latency 70ms
peakrate 1mbit minburst 1540
(yes that does 0.5mbit limiting, but it would be a start...)

#tc qdisc add dev eth0 root handle 1: htb default 20
#tc class add dev eth0 parent 1: classid 1:1 htb rate 1920kbit
#tc class add dev eth0 parent 1:1 classid 1:10 htb rate 128kbit prio 1
#tc class add dev eth0 parent 1:1 classid 1:20 htb rate 1792kbit prio 2
#tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
#tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
#tc filter add dev eth0 parent 1:0 protocol ip prio 10 u32 match ip tos
0x10 0xff flowid 1:10
#tc filter add dev eth0 parent 1:0 protocol ip prio 10 u32 match ip
protocol 1 0xff flowid 1:10

I use the latest iproute2 from the gentoo portage, marked as stable on
amd64:
sys-apps/iproute2-2.6.16.20060323/

I manually setup to NIC at server boot, with:
#!/bin/sh
mii-tool -F 10baseT-FD eth0
(but I had the same problems before I did that when the box was in a
private LAN)

The server details are:
(serenity:~) # uname -a89zx
Linux serenity 2.6.18-gentoo-r4 #1 SMP Mon Jan 15 12:33:56 CET 2007
x86_64 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux
(this is genkernel generated)

(serenity:~) # grep -i config_hz
/etc/kernels/kernel-config-x86_64-2.6.18-gentoo-r4
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250

(serenity:~) # lspci | grep -i ether
05:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5714
Gigabit Ethernet (rev a3)
05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714
Gigabit Ethernet (rev a3)

(serenity:~) # lsmod
Module                  Size  Used by
sch_sfq                 7424  0
sch_htb                18944  1
act_police              8480  1
cls_u32                 9480  2
sch_ingress             5248  1
sch_tbf                 8320  0
ipt_LOG                 8192  1
xt_limit                4352  1
xt_state                3840  1
ip_conntrack           53092  1 xt_state
xt_tcpudp               4864  5
iptable_filter          4736  1
iptable_mangle          4480  0
ip_tables              22312  2 iptable_filter,iptable_mangle
x_tables               18824  5
ipt_LOG,xt_limit,xt_state,xt_tcpudp,ip_tables

(serenity:~) # iptables -nL
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:20
flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:21
flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80
flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
dpts:20000:20100 flags:0x17/0x02
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state
RELATED,ESTABLISHED
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           icmp type 8
LOG        all  --  0.0.0.0/0            0.0.0.0/0           limit: avg
3/hour burst 5 LOG flags 0 level 4 prefix `iptables-IN-policy: ''

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

(the mangle tables are empty)

Regards,

-- 
Roman Skuła  (http://roman.skula.com)

Roman Skula wrote:
> 05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714
> Gigabit Ethernet (rev a3)
You may need to turn off segmentation offload with ethtool -k

Andy.

Andy Furniss napisał(a):
> Roman Skula wrote:
>> 05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714
>> Gigabit Ethernet (rev a3)
> You may need to turn off segmentation offload with ethtool -k
A huge, wet kiss for you, this turned my sky blue again. :)

Many thanks!

Best of wishes,

-- 
Roman Skuła  (http://roman.skula.com)

Hi there,
>> 05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714
>> Gigabit Ethernet (rev a3)
>You may need to turn off segmentation offload with ethtool -k
That''s very interesting. Do you have any more information on this
subject? Does tso in general distrupt accuracy of tc ops or is it the
case only with Broadcom nics? Any links would be very much appreciated.

Thanks for response in advance!

cheers,
Marek Kierdelewicz

Marek Kierdelewicz wrote:
> Hi there,
> 
> 
>>>05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714
>>>Gigabit Ethernet (rev a3)
> 
> 
>>You may need to turn off segmentation offload with ethtool -k
> 
> 
> That''s very interesting. Do you have any more information on this
> subject? Does tso in general distrupt accuracy of tc ops or is it the
> case only with Broadcom nics? Any links would be very much appreciated.
I don''t have any gig eth, but it''s not just Broadcom.

There have been other posts on here where people have needed to increase 
  htb''s mtu parameter to get htb to handle it - despite the interface 
being 1500. For shaping traffic for a slow link that is not going to be 
nice - you could end up dropping multiple tcp segments at once, and it 
will hurt jitter. I don''t imagine it''s right to do it for
shaping at gig
speed either, maybe the default long queue on eth saves dropping.

I suppose those using gig eth and shaping routed traffic at 1500 MTU 
will not notice unless they try and shape locally generated aswell.

Andy.

Roman Skula wrote:
> Andy Furniss napisał(a):
> 
>>Roman Skula wrote:
>>
>>>05:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5714
>>>Gigabit Ethernet (rev a3)
>>
>>You may need to turn off segmentation offload with ethtool -k
> 
> A huge, wet kiss for you, this turned my sky blue again. :)
Lol - I forgot to mention: be carefull about using htb default, your 
arp/other link layer end up there. If you don''t specify a default htb 
will let them through unshaped, you can make a filter to catch the 
unclassified IP traffic and send it to the class you want.

Andy.

Hello Andy,
How do one create such a filter to catch arp/other link layer traffic? Can
you give us one such example?

Thanks,
-nik

----- Original Message ----- 
From: "Andy Furniss" <lists@andyfurniss.entadsl.com>
To: "Roman Skula" <roman@skula.com>
Cc: <lartc@mailman.ds9a.nl>
Sent: Wednesday, January 17, 2007 10:18 PM
Subject: Re: [LARTC] egress bandwidth not limited / limitedextremely
inaccurately

> Roman Skula wrote:
> > Andy Furniss napisał(a):
> >
> >>Roman Skula wrote:
> >>
> >>>05:04.1 Ethernet controller: Broadcom Corporation NetXtreme
BCM5714
> >>>Gigabit Ethernet (rev a3)
> >>
> >>You may need to turn off segmentation offload with ethtool -k
> >
> > A huge, wet kiss for you, this turned my sky blue again. :)
>
> Lol - I forgot to mention: be carefull about using htb default, your
> arp/other link layer end up there. If you don''t specify a default
htb
> will let them through unshaped, you can make a filter to catch the
> unclassified IP traffic and send it to the class you want.
>
> Andy.
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>

Nikolay Kichukov wrote:
> Hello Andy,
> How do one create such a filter to catch arp/other link layer traffic? Can
> you give us one such example?
A quick test on ingress

#tc qdisc add dev eth0 ingress

Classify all ip traffic

#tc filter add dev eth0 parent ffff: prio 1 protocol ip u32 match u32 0 
0 flowid :1

All arp

#tc filter add dev eth0 parent ffff: prio 2 protocol arp u32 match u32 0 
0 flowid :2

Anything else

#tc filter add dev eth0 parent ffff: prio 3 protocol all u32 match u32 0 
0 flowid :3

Look at the counters

#tc -s filter ls dev eth0 parent ffff:

Delete everything ingress on eth0

#tc qdisc del dev eth0 ingress


You can use ethertype protocol numbers in place of arp/ip.
Use the prio to make sure the catch alls are last in the filters you use 
- prio 1 is the highest for filters.

Andy.

Cheers Andy,
That clarifies;-)

Regards,
-nik

Andy Furniss wrote:
> Nikolay Kichukov wrote:
>> Hello Andy,
>> How do one create such a filter to catch arp/other link layer traffic? 
>> Can
>> you give us one such example?
> 
> A quick test on ingress
> 
> #tc qdisc add dev eth0 ingress
> 
> Classify all ip traffic
> 
> #tc filter add dev eth0 parent ffff: prio 1 protocol ip u32 match u32 0 
> 0 flowid :1
> 
> All arp
> 
> #tc filter add dev eth0 parent ffff: prio 2 protocol arp u32 match u32 0 
> 0 flowid :2
> 
> Anything else
> 
> #tc filter add dev eth0 parent ffff: prio 3 protocol all u32 match u32 0 
> 0 flowid :3
> 
> Look at the counters
> 
> #tc -s filter ls dev eth0 parent ffff:
> 
> Delete everything ingress on eth0
> 
> #tc qdisc del dev eth0 ingress
> 
> 
> You can use ethertype protocol numbers in place of arp/ip.
> Use the prio to make sure the catch alls are last in the filters you use 
> - prio 1 is the highest for filters.
> 
> Andy.
>