LARTC - Sep 2007 - major packet loss with htb

Hi,

got a 25/25Mbit connection which is quite stuffed. So I applied htb rules.

Uplink:

class htb 1:1 root rate 24500Kbit ceil 24500Kbit burst 4661b/8 mpu 0b 
overhead 0b cburst 4661b/8 mpu 0b overhead 0b level 7
  Sent 430600689269 bytes 730147320 pkt (dropped 0, overlimits 0 requeues 0)
  rate 23057Kbit 5520pps backlog 0b 0p requeues 0
  lended: 199673949 borrowed: 0 giants: 0
  tokens: -964 ctokens: -964

class htb 1:10 parent 1:1 leaf 10: prio 1 quantum 93750 rate 7500Kbit 
ceil 10000Kbit burst 2536b/8 mpu 0b overhead 0b cburst 2850b/8 mpu 0b 
overhead 0b level 0
  Sent 36514538104 bytes 122225933 pkt (dropped 250, overlimits 0 
requeues 0)
  rate 2933Kbit 1236pps backlog 0b 0p requeues 0
  lended: 117912267 borrowed: 4313666 giants: 0
  tokens: 2485 ctokens: 2114

class htb 1:20 parent 1:1 leaf 20: prio 3 quantum 125000 rate 10000Kbit 
ceil 20000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b 
overhead 0b level 0
  Sent 328942394630 bytes 480708749 pkt (dropped 1725703, overlimits 0 
requeues 0)
  rate 17344Kbit 3494pps backlog 0b 58p requeues 0
  lended: 290657995 borrowed: 190050689 giants: 0
  tokens: -2409 ctokens: -1914

class htb 1:30 parent 1:1 leaf 30: prio 5 quantum 37500 rate 3000Kbit 
ceil 3100Kbit burst 1974b/8 mpu 0b overhead 0b cburst 1986b/8 mpu 0b 
overhead 0b level 0
  Sent 66686833287 bytes 127212829 pkt (dropped 3728008, overlimits 0 
requeues 0)
  rate 3099Kbit 792pps backlog 0b 126p requeues 0
  lended: 121903109 borrowed: 5309594 giants: 0
  tokens: -6323 ctokens: -6993


1:10 is icmp, interactive and http, https
1:30 is P2P as detected by ipp2p
1:20 is the rest

As you can see 1:10 hasn''t even dropped a packet. Though when I ping
the
our Internet gateway I get terrible packet loss, at least 10%. But why, 
I don''t see any packets dropped in 1:10?

Downlink:

class htb 2:1 root rate 1000Mbit ceil 1000Mbit burst 126500b/8 mpu 0b 
overhead 0b cburst 126500b/8 mpu 0b overhead 0b level 7
  Sent 510719706792 bytes 662575747 pkt (dropped 0, overlimits 0 requeues 0)
  rate 25671Kbit 4333pps backlog 0b 0p requeues 0
  lended: 4926904 borrowed: 0 giants: 0
  tokens: 1012 ctokens: 1012

class htb 2:20 parent 2:1 prio 1 quantum 200000 rate 900000Kbit ceil 
1000Mbit burst 114075b/8 mpu 0b overhead 0b cburst 126500b/8 mpu 0b 
overhead 0b level 0
  Sent 443914952356 bytes 553603614 pkt (dropped 0, overlimits 0 requeues 0)
  rate 22555Kbit 3692pps backlog 0b 0p requeues 0
  lended: 553603614 borrowed: 0 giants: 0
  tokens: 1014 ctokens: 1012

class htb 2:30 parent 2:1 prio 5 quantum 37500 rate 3000Kbit ceil 
3100Kbit burst 1974b/8 mpu 0b overhead 0b cburst 1986b/8 mpu 0b overhead 
0b level 0
  Sent 66805273059 bytes 108972939 pkt (dropped 8183735, overlimits 0 
requeues 0)
  rate 3128Kbit 647pps backlog 0b 806p requeues 0
  lended: 104045229 borrowed: 4926904 giants: 0
  tokens: -6997 ctokens: -7235

1:20 is everything except P2P
1:30 is P2P as detected by ipp2p

I know I should be using ingress here, but I''d like to get the uplink 
working first and this still helps throttle the P2P down. Also notice 
that 2:20 hasn''t dropped any packets.

I have only specified prios which as I understand don''t really matter 
ATM. I haven''t touched quantums nor bursts, only rates and ceils. But I
don''t think these problems are caused by this alone. What am I missing?

Also with the current configuration uplink nor downlink go near 25Mbit, 
so I there should be free bandwidth.

The qos commands are as follows:

RATE_TOTAL="24500kbit"
QUANTUM_BIG="20000"
RATE_HIGH="7500kbit"
CEIL_HIGH="10000kbit"
RATE_DEFAULT="10000kbit"
CEIL_DEFAULT="20000kbit"
RATE_P2P="3000kbit"
CEIL_P2P="3100kbit"

tc qdisc add dev eth1 root handle 1:0 htb default 20
tc qdisc add dev eth0 root handle 2:0 htb default 20

tc class add dev eth1 parent 1:0 classid 1:1 htb rate $RATE_TOTAL ceil 
$RATE_TOTAL
tc class add dev eth0 parent 2:0 classid 2:1 htb rate 1Gbit ceil 1Gbit 
quantum $QUANTUM_BIG

tc class add dev eth1 parent 1:1 classid 1:10 htb rate $RATE_HIGH ceil 
$CEIL_HIGH prio 1
tc class add dev eth1 parent 1:1 classid 1:20 htb rate $RATE_DEFAULT 
ceil $CEIL_DEFAULT prio 3
tc class add dev eth1 parent 1:1 classid 1:30 htb rate $RATE_P2P ceil 
$CEIL_P2P prio 5

tc class add dev eth0 parent 2:1 classid 2:20 htb rate 900Mbit ceil 
1Gbit prio 1
tc class add dev eth0 parent 2:1 classid 2:30 htb rate $RATE_P2P ceil 
$CEIL_P2P prio 5

tc qdisc add dev eth1 parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev eth1 parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev eth1 parent 1:30 handle 30: sfq perturb 10


Please point me in the right directions.

Thanks,


-- 
Aleksander Kamenik
system administrator
+372 6659 649
aleksander@krediidiinfo.ee

Krediidiinfo AS
http://www.krediidiinfo.ee/

Got a solution!

So the problem is, when you have enough rate specified, not all of it is 
used up, but still have shitty latency and packet loss for a class, is 
that the ISP screwed up.

Their device is either doing 25/10Mbit (dl/ul) instead of advertised 
25/25Mbit or the router is simply coughing blood when more than 10Mbit 
is being uploaded (borked).

Simply said, there wasn''t enough bandwidth to begin with. I was shaping
outgoing bandwidth to 25Mbit, but that was as pointless as if I had 
shaped it to 50Mbit. There wasn''t enough anyway.

Bringing it down to less than 10Mbit solved the problem. Next step is to 
make the ISP fix their shaping.


Regards,

-- 
Aleksander Kamenik
system administrator
+372 6659 649
aleksander@krediidiinfo.ee

Krediidiinfo AS
http://www.krediidiinfo.ee/