LARTC - Nov 2006 - Why did I need strange ceiling settings? (full version)

Sorry I pressed the wrong key and sent the message too early...
> -----Ursprüngliche Nachricht-----
> Von: Philipp Leusmann [mailto:philipp.leusmann@rwth-aachen.de]
> Gesendet: Mittwoch, 8. November 2006 12:53
> An: ''lartc@mailman.ds9a.nl''
> Betreff: Why did I need strange ceiling settings?
> 
> Hi all,
> 
> I recently installed traffic shaping on my ADSL line with a nominal upload
> rate of 1MBit. My Modem says it has an upload bitrate of 843 kbits.
> So I thought, to use a ceiling of 800kbit for the root qdisc is a good
> idea.
> But with that setting I only achieved upload rates of around 300kbits
> whereas I reached around 650 kbits without traffic shaping.
> Yesterday I played a little with the ceiling value and found that
> increasing it to 175kBps did the job.
> Could anybody please explain this strange behaviour?
> 
> My shaping script looks as follows:
UPRATE="175kbps"
P2PRATE="20kbps"
PRIORATE1="80kbps"
PRIORATE2="50kbps"
PRIORATE3="40kbps"
PRIORATE4="5kbps" 

MTU="`/sbin/ifconfig $EXTIF | grep ''MTU'' | awk
''{print $6}'' | sed -e
''s/.*://''`"

# Quantum
QUANTUM1=$(($MTU*4))
QUANTUM2=$(($MTU*3))
QUANTUM3=$(($MTU*2))
QUANTUM4=$MTU

# Burst
## removed bursts
#BURST1="6k"
#BURST2="4k"
#BURST3="2k"
#BURST4="0k"
#CBURST1="3k"
#CBURST2="2k"
#CBURST3="1k"
#CBURST4="0k"

echo "Set queue length for IFACE"
# Set queue length for IFACE
ifconfig $IFACE txqueuelen 16

echo "Trying to delete old ruleset. Will give error if it does not
exist"
tc qdisc del dev $IFACE root 

echo "Specify queue discipline"
# Specify queue discipline
tc qdisc add dev $IFACE root handle 1:0 htb default 103

echo "Set root class"
# Set root class
tc class add dev $IFACE parent 1:0 classid 1:1 htb rate $UPRATE ceil $UPRATE
echo "Specify sub classes"
# Specify sub classes
tc class add dev $IFACE parent 1:1 classid 1:101 htb rate $PRIORATE1 ceil
$UPRATE quantum $QUANTUM1 prio 0
tc class add dev $IFACE parent 1:1 classid 1:102 htb rate $PRIORATE2 ceil
$UPRATE quantum $QUANTUM2 prio 1
tc class add dev $IFACE parent 1:1 classid 1:103 htb rate $PRIORATE3 ceil
$UPRATE quantum $QUANTUM3 prio 2
tc class add dev $IFACE parent 1:1 classid 1:104 htb rate $PRIORATE4 ceil
$P2PRATE quantum $QUANTUM4 prio 3

echo "Filter packets"
# Filter packets
tc filter add dev $IFACE parent 1:0 protocol ip prio 0 handle $MARKPRIO1 fw
classid 1:101
tc filter add dev $IFACE parent 1:0 protocol ip prio 1 handle $MARKPRIO2 fw
classid 1:102
tc filter add dev $IFACE parent 1:0 protocol ip prio 2 handle $MARKPRIO3 fw
classid 1:103
tc filter add dev $IFACE parent 1:0 protocol ip prio 3 handle $MARKPRIO4 fw
classid 1:104

UPRATE="175kbps"
P2PRATE="20kbps"
PRIORATE1="80kbps"
PRIORATE2="50kbps"
PRIORATE3="40kbps"
PRIORATE4="5kbps" 

MTU="`/sbin/ifconfig $EXTIF | grep ''MTU'' | awk
''{print $6}'' | sed -e
''s/.*://''`"

# Quantum
QUANTUM1=$(($MTU*4))
QUANTUM2=$(($MTU*3))
QUANTUM3=$(($MTU*2))
QUANTUM4=$MTU

# Burst
## removed bursts
#BURST1="6k"
#BURST2="4k"
#BURST3="2k"
#BURST4="0k"
#CBURST1="3k"
#CBURST2="2k"
#CBURST3="1k"
#CBURST4="0k"

echo "Set queue length for IFACE"
# Set queue length for IFACE
ifconfig $IFACE txqueuelen 16

echo "Trying to delete old ruleset. Will give error if it does not
exist"
tc qdisc del dev $IFACE root 

echo "Specify queue discipline"
# Specify queue discipline
tc qdisc add dev $IFACE root handle 1:0 htb default 103

echo "Set root class"
# Set root class
tc class add dev $IFACE parent 1:0 classid 1:1 htb rate $UPRATE ceil $UPRATE
echo "Specify sub classes"
# Specify sub classes
tc class add dev $IFACE parent 1:1 classid 1:101 htb rate $PRIORATE1 ceil
$UPRATE quantum $QUANTUM1 prio 0
tc class add dev $IFACE parent 1:1 classid 1:102 htb rate $PRIORATE2 ceil
$UPRATE quantum $QUANTUM2 prio 1
tc class add dev $IFACE parent 1:1 classid 1:103 htb rate $PRIORATE3 ceil
$UPRATE quantum $QUANTUM3 prio 2
tc class add dev $IFACE parent 1:1 classid 1:104 htb rate $PRIORATE4 ceil
$P2PRATE quantum $QUANTUM4 prio 3

echo "Filter packets"
# Filter packets
tc filter add dev $IFACE parent 1:0 protocol ip prio 0 handle $MARKPRIO1 fw
classid 1:101
tc filter add dev $IFACE parent 1:0 protocol ip prio 1 handle $MARKPRIO2 fw
classid 1:102
tc filter add dev $IFACE parent 1:0 protocol ip prio 2 handle $MARKPRIO3 fw
classid 1:103
tc filter add dev $IFACE parent 1:0 protocol ip prio 3 handle $MARKPRIO4 fw
classid 1:104




Thanks in advance and sorry for the incomplete posting.

Greetings,
 Philipp

Philipp Leusmann wrote:

You will need to back off from the rates more and use kbit of course.
> tc qdisc add dev $IFACE root handle 1:0 htb default 103
default is bad if $IFACE is eth your arp will go here, also if eth 
Quantum should be set to ip mtu + 14.
> 
> echo "Set root class"
> # Set root class
> tc class add dev $IFACE parent 1:0 classid 1:1 htb rate $UPRATE ceil
$UPRATE
> echo "Specify sub classes"
> # Specify sub classes
> tc class add dev $IFACE parent 1:1 classid 1:101 htb rate $PRIORATE1 ceil
> $UPRATE quantum $QUANTUM1 prio 0
> tc class add dev $IFACE parent 1:1 classid 1:102 htb rate $PRIORATE2 ceil
> $UPRATE quantum $QUANTUM2 prio 1
> tc class add dev $IFACE parent 1:1 classid 1:103 htb rate $PRIORATE3 ceil
> $UPRATE quantum $QUANTUM3 prio 2
> tc class add dev $IFACE parent 1:1 classid 1:104 htb rate $PRIORATE4 ceil
> $P2PRATE quantum $QUANTUM4 prio 3
Using different quantums makes more sense if classes have the same prio 
- like this the higher prio classes get all spare anyway if they need it.
> 
> echo "Filter packets"
> # Filter packets
> tc filter add dev $IFACE parent 1:0 protocol ip prio 0 handle $MARKPRIO1 fw
> classid 1:101
> tc filter add dev $IFACE parent 1:0 protocol ip prio 1 handle $MARKPRIO2 fw
> classid 1:102
> tc filter add dev $IFACE parent 1:0 protocol ip prio 2 handle $MARKPRIO3 fw
> classid 1:103
> tc filter add dev $IFACE parent 1:0 protocol ip prio 3 handle $MARKPRIO4 fw
> classid 1:104
Makes no difference as such in this case, but highest prio for filters is 1.

Andy.

> -----Ursprüngliche Nachricht-----
> Von: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl]
> Im Auftrag von Andy Furniss
> Gesendet: Sonntag, 12. November 2006 21:51
> An: Philipp Leusmann
> Cc: lartc@mailman.ds9a.nl
> Betreff: Re: [LARTC] Why did I need strange ceiling settings? (full
> version)
> 
> Philipp Leusmann wrote:
> 
> You will need to back off from the rates more and use kbit of course.
> 
> > tc qdisc add dev $IFACE root handle 1:0 htb default 103
> 
> default is bad if $IFACE is eth your arp will go here, also if eth
> Quantum should be set to ip mtu + 14.
$IFACE is ppp0. Does this make difference?
And you would recommend to use no backup at all?

Regards,
 Philipp

Philipp Leusmann wrote:
> 
>>-----Ursprüngliche Nachricht-----
>>Von: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl]
>>Im Auftrag von Andy Furniss
>>Gesendet: Sonntag, 12. November 2006 21:51
>>An: Philipp Leusmann
>>Cc: lartc@mailman.ds9a.nl
>>Betreff: Re: [LARTC] Why did I need strange ceiling settings? (full
>>version)
>>
>>Philipp Leusmann wrote:
>>
>>You will need to back off from the rates more and use kbit of course.
>>
>>
>>>tc qdisc add dev $IFACE root handle 1:0 htb default 103
>>
>>default is bad if $IFACE is eth your arp will go here, also if eth
>>Quantum should be set to ip mtu + 14.
> 
> 
> $IFACE is ppp0. Does this make difference?
Yes - using htb default is safe on ppp and quantum doesn''t need 14 
adding. One caveat, if you get ppp MTU by script what if mtu on ppp is 
really big - my old ISP used to ask (during ppp negotiation) for mru of 
about 32k (aal5 mtu), which meant that mtu on the ppp was set to 32k.
> And you would recommend to use no backup at all?
Had it been eth then you could have made a catch all ip filter with 
lower prio to get anything else. You could also have made a filter for 
arp/other non ip - but if non ip trafic levels are low I would just let 
them through unshaped, which is what htb does if you don''t specify a 
default class / use default 0. (hfsc is the opposite - unclassified gets 
dropped by default).

Try setting uprate ceil to 600kbit and make sure the sum of rates 
doesn''t exceed it.

Upload for a few minutes and while still uploading do -

tc -s -d class ls dev ppp0

and post the output.

Andy.


post the output

Hey all:

I want to limit incoming traffic to around 300 packets per second.  When
I receive higher amounts than that it kills my processor and causes
problems with UML.  Can anyone point me to a way to do this?  Thanks a
lot. 

Jon

All I''m a little confused about something. 

I''m using GRED and HTB deployed in a AF/EF terms of service
environment.
I am rate limiting to 550 kbps.  If I take GRED off my leaf HTB classes
then FIFO will take over.  The problem I see is that my rate received
goes up to 700 kbps rather than 550 with GRED.  

Questions:

My question is what is causing the rate limit?  HTB or GRED/FIFO
	- Does HTB say start dropping... and then GRED or FIFO is the
method at which to drop?

Is GRED more accurate because it is dropping before the rate limit is
breached?  (early drop)

Is FIFO higher because it drops after the rate has been breached?

Hi andy,

I reset the ceiling to 600kbit and get same same bad results as before. Also
I set all classes to use the same quantum which is mtu (it is 1488 here).
Here is the output you requested:

miles:~# tc -s -d class ls dev ppp0
class htb 1:101 parent 1:1 leaf 8019: prio 0 quantum 1488 rate 150000bit
ceil 600000bit burst 1674b/8 mpu 0b overhead 0b cburst 1899b/8 mpu 0b
overhead 0b level 0 
 Sent 130659 bytes 806 pkts (dropped 0, overlimits 0) 
 rate 224bit 
 lended: 632 borrowed: 174 giants: 0
 tokens: 84164 ctokens: 24117

class htb 1:1 root rate 600000bit ceil 600000bit burst 1899b/8 mpu 0b
overhead 0b cburst 1899b/8 mpu 0b overhead 0b level 7 
 Sent 27239843 bytes 29309 pkts (dropped 0, overlimits 0) 
 rate 286640bit 34pps 
 lended: 16484 borrowed: 0 giants: 0
 tokens: -66101 ctokens: -66101

class htb 1:103 parent 1:1 leaf 801b: prio 2 quantum 1488 rate 250000bit
ceil 600000bit burst 1724b/8 mpu 0b overhead 0b cburst 1899b/8 mpu 0b
overhead 0b level 0 
 Sent 27111784 bytes 28505 pkts (dropped 0, overlimits 0) 
 rate 286232bit 34pps backlog 2p 
 lended: 12193 borrowed: 16310 giants: 0
 tokens: -83395 ctokens: -41934

class htb 1:102 parent 1:1 leaf 801a: prio 1 quantum 1488 rate 150000bit
ceil 600000bit burst 1674b/8 mpu 0b overhead 0b cburst 1899b/8 mpu 0b
overhead 0b level 0 
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0) 
 lended: 0 borrowed: 0 giants: 0
 tokens: 91601 ctokens: 25976

class htb 1:104 parent 1:1 leaf 801c: prio 3 quantum 1488 rate 50000bit ceil
200000bit burst 1624b/8 mpu 0b overhead 0b cburst 1699b/8 mpu 0b overhead 0b
level 0 
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0) 
 lended: 0 borrowed: 0 giants: 0
 tokens: 266600 ctokens: 69726

I hope this helps to track down the problem.

Thanks,
 Philipp
> -----Ursprüngliche Nachricht-----
> Von: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl]
> Im Auftrag von Andy Furniss
> Gesendet: Dienstag, 14. November 2006 00:50
> An: Philipp Leusmann
> Cc: lartc@mailman.ds9a.nl
> Betreff: Re: AW: [LARTC] Why did I need strange ceiling settings? (full
> version)
> 
> Philipp Leusmann wrote:
> >
> >>-----Ursprüngliche Nachricht-----
> >>Von: lartc-bounces@mailman.ds9a.nl [mailto:lartc-
> bounces@mailman.ds9a.nl]
> >>Im Auftrag von Andy Furniss
> >>Gesendet: Sonntag, 12. November 2006 21:51
> >>An: Philipp Leusmann
> >>Cc: lartc@mailman.ds9a.nl
> >>Betreff: Re: [LARTC] Why did I need strange ceiling settings? (full
> >>version)
> >>
> >>Philipp Leusmann wrote:
> >>
> >>You will need to back off from the rates more and use kbit of
course.
> >>
> >>
> >>>tc qdisc add dev $IFACE root handle 1:0 htb default 103
> >>
> >>default is bad if $IFACE is eth your arp will go here, also if eth
> >>Quantum should be set to ip mtu + 14.
> >
> >
> > $IFACE is ppp0. Does this make difference?
> 
> Yes - using htb default is safe on ppp and quantum doesn''t need 14
> adding. One caveat, if you get ppp MTU by script what if mtu on ppp is
> really big - my old ISP used to ask (during ppp negotiation) for mru of
> about 32k (aal5 mtu), which meant that mtu on the ppp was set to 32k.
> 
> > And you would recommend to use no backup at all?
> 
> Had it been eth then you could have made a catch all ip filter with
> lower prio to get anything else. You could also have made a filter for
> arp/other non ip - but if non ip trafic levels are low I would just let
> them through unshaped, which is what htb does if you don''t specify
a
> default class / use default 0. (hfsc is the opposite - unclassified gets
> dropped by default).
> 
> Try setting uprate ceil to 600kbit and make sure the sum of rates
> doesn''t exceed it.
> 
> Upload for a few minutes and while still uploading do -
> 
> tc -s -d class ls dev ppp0
> 
> and post the output.
> 
> Andy.
> 
> 
> post the output
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Philipp Leusmann wrote:
> Hi andy,
> 
> I reset the ceiling to 600kbit and get same same bad results as before.
Also
> I set all classes to use the same quantum which is mtu (it is 1488 here).
> Here is the output you requested:
> 
> class htb 1:103 parent 1:1 leaf 801b: prio 2 quantum 1488 rate 250000bit
> ceil 600000bit burst 1724b/8 mpu 0b overhead 0b cburst 1899b/8 mpu 0b
> overhead 0b level 0 
>  Sent 27111784 bytes 28505 pkts (dropped 0, overlimits 0) 
>  rate 286232bit 34pps backlog 2p 
>  lended: 12193 borrowed: 16310 giants: 0
>  tokens: -83395 ctokens: -41934
That is strange - assuming that upload is tcp, there are no drops and 
only a backlog of 2.

When uploading you are somewhat dependant on the size of the window 
advertised by the far end - also loss, which in this case is not by htb, 
will make your cwind small.

I think what you need to do is tcpdump -s 100 -w dumpfile the connection 
from the start and have a look/ post what''s going on.

Another way to see if it''s external loss is on the sender do a few

netstat -s | grep retrans

and look at the counter.

Small chance there could be some window scaling mismatch caused by a 
broken router in the way.

The loss could be isp/target server dropping or -

You mentioned a nominal upload rate of 1mbit which you don''t reach. If 
you are synced at a low target SNR margin then some modems will doggedly 
hold the line and take the loss - others will drop and resync (often at 
a similar rate as the extra noise that causes the resync is gone when it 
retrains). I have to limit my downrate to 75% of 6db speed and it still 
drops sometimes. My up is solid, though, as it''s limited to 50% due to 
the product I am on (448/up to 8128 - horribly asymmetric if I could 
sync at 8128)

As to why shaping on/off makes such a difference - I am not sure, you 
are backlogged so there is some limiting happening, so maybe the higher 
speeds achieved without htb rely on being able to burst out full speed 
whenever loss is low.

Andy.

Hi Andy,

I made the dumpfile and will send it to you in a separate private email. I
cannot see anything suspicious, but maybe I am not looking for the right
thing.
Same goes for netstat -s | grep retrans : The count does not rise during the
transfer.
For the modem, here is what it says:
                                down         up
Bit-rate  (fast)          :       15694         915
Bit-rate  (relative cap.) :       100 %       100 %
Bit-rate  (max)           :       15694         915
FEC error (fast)          :        7116           0
CRC error (fast)          :       13421           0
HEC error (fast)          :        5051           0
Noise margin              :      8.3 dB      8.5 dB 
Attenuation               :     16.0 dB     12.8 dB 
Transmit power            :     22.3 dBm    12.3 dBm
First channel             :          64          33
Last channel              :         505          59
Channel gaps              : 95 110 127 188 191 243 291 348

All this is very irritating. I don´t think it depends on the remote host,
because it appears on every remote I try. 
Are you sure that tc makes everything right? I would say it’s the easiest
point of failure. 
Do you have any more ideas?

Thanks,
 Philipp
> -----Ursprüngliche Nachricht-----
> Von: lartc-bounces@mailman.ds9a.nl [mailto:lartc-bounces@mailman.ds9a.nl]
> Im Auftrag von Andy Furniss
> Gesendet: Dienstag, 14. November 2006 20:39
> An: Philipp Leusmann
> Cc: lartc@mailman.ds9a.nl
> Betreff: Re: AW: AW: [LARTC] Why did I need strange ceiling settings?
> (fullversion)
> 
> Philipp Leusmann wrote:
> > Hi andy,
> >
> > I reset the ceiling to 600kbit and get same same bad results as
before.
> Also
> > I set all classes to use the same quantum which is mtu (it is 1488
> here).
> > Here is the output you requested:
> 
> >
> > class htb 1:103 parent 1:1 leaf 801b: prio 2 quantum 1488 rate
250000bit
> > ceil 600000bit burst 1724b/8 mpu 0b overhead 0b cburst 1899b/8 mpu 0b
> > overhead 0b level 0
> >  Sent 27111784 bytes 28505 pkts (dropped 0, overlimits 0)
> >  rate 286232bit 34pps backlog 2p
> >  lended: 12193 borrowed: 16310 giants: 0
> >  tokens: -83395 ctokens: -41934
> 
> That is strange - assuming that upload is tcp, there are no drops and
> only a backlog of 2.
> 
> When uploading you are somewhat dependant on the size of the window
> advertised by the far end - also loss, which in this case is not by htb,
> will make your cwind small.
> 
> I think what you need to do is tcpdump -s 100 -w dumpfile the connection
> from the start and have a look/ post what''s going on.
> 
> Another way to see if it''s external loss is on the sender do a few
> 
> netstat -s | grep retrans
> 
> and look at the counter.
> 
> Small chance there could be some window scaling mismatch caused by a
> broken router in the way.
> 
> The loss could be isp/target server dropping or -
> 
> You mentioned a nominal upload rate of 1mbit which you don''t
reach. If
> you are synced at a low target SNR margin then some modems will doggedly
> hold the line and take the loss - others will drop and resync (often at
> a similar rate as the extra noise that causes the resync is gone when it
> retrains). I have to limit my downrate to 75% of 6db speed and it still
> drops sometimes. My up is solid, though, as it''s limited to 50%
due to
> the product I am on (448/up to 8128 - horribly asymmetric if I could
> sync at 8128)
> 
> As to why shaping on/off makes such a difference - I am not sure, you
> are backlogged so there is some limiting happening, so maybe the higher
> speeds achieved without htb rely on being able to burst out full speed
> whenever loss is low.
> 
> Andy.
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Philipp Leusmann wrote:
> Hi Andy,
> 
> I made the dumpfile and will send it to you in a separate private email. I
> cannot see anything suspicious, but maybe I am not looking for the right
> thing.
Did you get my offlist reply about that?
> Same goes for netstat -s | grep retrans : The count does not rise during
the
> transfer.
> For the modem, here is what it says:
>                                 down         up
> Bit-rate  (fast)          :       15694         915
> Bit-rate  (relative cap.) :       100 %       100 %
> Bit-rate  (max)           :       15694         915
> FEC error (fast)          :        7116           0
> CRC error (fast)          :       13421           0
> HEC error (fast)          :        5051           0
> Noise margin              :      8.3 dB      8.5 dB 
> Attenuation               :     16.0 dB     12.8 dB 
> Transmit power            :     22.3 dBm    12.3 dBm
> First channel             :          64          33
> Last channel              :         505          59
> Channel gaps              : 95 110 127 188 191 243 291 348
0 upstream errors may mean your modem isn''t reporting/getting them 
properly from the far end, but if the uptime of those stats isn''t too 
short, the errors don''t look too bad anyway.

As for what I said about showtime rate being multiple of 32kbit I was 
thinking adsl, I guess it''s different for adsl2(+), I suppose the FEC 
overheads could be deducted first aswell (We don''t get FEC on fast in 
the UK)

Andy.