I thought I had this all worked out, but it seems not. The following tc
configuration:
tc qdisc del dev ppp0 root 2> /dev/null > /dev/null
tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1
tc qdisc add dev ppp0 parent 1:1 handle 2: prio bands 3
tc qdisc add dev ppp0 parent 2:1 handle 10: sfq perturb 20
tc qdisc add dev ppp0 parent 2:2 handle 20: sfq perturb 20
tc qdisc add dev ppp0 parent 2:3 handle 30: sfq perturb 20
# ICMP so we can see PRIO working
tc filter add dev ppp0 parent 2:0 protocol ip prio 10 u32 match ip protocol 1
0xff flowid 2:1
Is producing the following tc rules:
qdisc tbf 1: rate 120000bit burst 1199b lat 4294.9s
Sent 2024323 bytes 38831 pkt (dropped 54, overlimits 280 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 2: parent 1:1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 2024323 bytes 38831 pkt (dropped 0, overlimits 0 requeues 280)
rate 0bit 0pps backlog 0b 0p requeues 280
qdisc sfq 10: parent 2:1 limit 128p quantum 1452b perturb 20sec
Sent 123362 bytes 1983 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 20: parent 2:2 limit 128p quantum 1452b perturb 20sec
Sent 122669 bytes 1007 pkt (dropped 0, overlimits 0 requeues 127)
rate 0bit 0pps backlog 0b 0p requeues 127
qdisc sfq 30: parent 2:3 limit 128p quantum 1452b perturb 20sec
Sent 1778292 bytes 35841 pkt (dropped 0, overlimits 0 requeues 153)
rate 0bit 0pps backlog 0b 0p requeues 153
And I have the following iptables rules:
Chain PREROUTING (policy ACCEPT 1564K packets, 875M bytes)
pkts bytes target prot opt in out source destination
1560K 872M tcpre all -- * * 0.0.0.0/0 0.0.0.0/0
Chain tcpre (1 references)
pkts bytes target prot opt in out source destination
1504K 864M CONNMARK tcp -- * * 0.0.0.0/0 0.0.0.0/0
CONNMARK restore mask 0xff
1280K 726M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0
MARK match !0x0/0xff
86 8988 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0
ipp2p v0.7.2 --bit MARK set 0x1
86 8988 CONNMARK tcp -- * * 0.0.0.0/0 0.0.0.0/0
MARK match 0x1/0xff CONNMARK save mask 0xff
Chain POSTROUTING (policy ACCEPT 1533K packets, 866M bytes)
pkts bytes target prot opt in out source destination
599K 29M CLASSIFY tcp -- * ppp0 0.0.0.0/0 0.0.0.0/0
tcp flags:0x16/0x10 length 0:128 CLASSIFY set 2:1
1529K 864M tcpost all -- * * 0.0.0.0/0 0.0.0.0/0
Chain tcpost (1 references)
pkts bytes target prot opt in out source destination
212K 111M CLASSIFY all -- * * 0.0.0.0/0 0.0.0.0/0
CLASSIFY set 2:2
549K 28M CLASSIFY all -- * ppp0 0.0.0.0/0 0.0.0.0/0
MARK match 0x1/0xff CLASSIFY set 2:3
713K 689M CLASSIFY all -- * eth0 0.0.0.0/0 0.0.0.0/0
MARK match 0x1/0xff CLASSIFY set 2:3
30518 2862K CLASSIFY all -- * * 10.75.22.1 66.1.2.3
CLASSIFY set 2:1
0 0 CLASSIFY all -- * * 66.1.2.3 66.9.8.7
CLASSIFY set 2:1
0 0 CLASSIFY udp -- * * 0.0.0.0/0 66.9.8.7
udp dpt:29378 CLASSIFY set 2:1
0 0 CLASSIFY udp -- * * 10.75.22.1 0.0.0.0/0
udp spt:29378 CLASSIFY set 2:1
But it seems that some outbound flows are being blocked entirely. I
don''t think they are being starved though. Even if they end up in 2:3,
they should at least be treated fairly. But I am producing a flow to
66.1.2.3 which does increment the counters in 2:1 but after a few
packets the flow stalls:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 8240 10.75.22.1:55318 66.1.2.3:922
ESTABLISHED 21028/ssh
And that flow stays stuck with 8240 bytes in the Send-Q.
What have I done wrong? I am so close to my desired behaviour with the
exception being of course that some flows seem to get stuck. I should
add that some initial packets in that stuck flow above (port 922) are
processed. It seems that at some point they just stop getting
processed. The last packets that went out on that flow were:
11:26:47.691755 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1872:1920(48) ack 2261
win 9968 <nop,nop,timestamp 8735287 43843955>
11:26:47.731489 IP 66.9.8.7.55318 > 66.1.2.3.922: . ack 1920 win 2548
<nop,nop,timestamp 43844116 8735287>
11:26:47.855781 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1920:1968(48) ack 2261
win 9968 <nop,nop,timestamp 8735300 43844116>
11:26:47.856142 IP 66.9.8.7.55318 > 66.1.2.3.922: . ack 1968 win 2548
<nop,nop,timestamp 43844240 8735300>
11:26:47.857866 IP 66.9.8.7.55318 > 66.1.2.3.922: P 2261:2373(112) ack 1968
win 2548 <nop,nop,timestamp 43844242 8735300>
11:26:48.072776 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1968:2016(48) ack 2373
win 9968 <nop,nop,timestamp 8735316 43844242>
11:26:48.321959 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1968:2016(48) ack 2373
win 9968 <nop,nop,timestamp 8735352 43844242>
11:26:48.322384 IP 66.9.8.7.55318 > 66.1.2.3.922: . ack 2016 win 2548
<nop,nop,timestamp 43844707 8735352,nop,nop,sack 1 {1968:2016}>
So maybe for some reason that last ack is not being dequeued? But
surely it should as classes 2:1 and 2:2 are virtually quiet and class
2:3 while having steady traffic, should at least treat it all fairly,
no?
b.
--
My other computer is your Microsoft Windows server.
Brian J. Murrell
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
On Sunday 04 December 2005 17:36, Brian J. Murrell wrote:> Even if they end up in 2:3, they should at least be treated fairly.2:3 will not be treated at all as long as 2:1 and 2:2 (which have higher priority) are occupied. If the queues in 2:1 and 2:2 resp. never empty, the packets in 2:3 will never be sent. There is no fair treatment in PRIO. That''s the whole purpose of this scheduler, to give one band of packets absolute priority over the other.> What have I done wrong?This is just out of personal interest, but could you try using instead of your TBF qdisc, a very simple HTB Qdisc / class with the same bandwidth limitation? If that solves the problem, then you''re suffering from a problem that I failed to solve when I last tried to use TBF; for some reason it got stuck on me too.> So maybe for some reason that last ack is not being dequeued?I don''t understand what you mean by fair treatment, but do try putting all ACKs into high priority band, then it will have to be dequeued. Regards, Andreas Klauer
On Sun, 2005-12-04 at 17:54 +0100, Andreas Klauer wrote:> On Sunday 04 December 2005 17:36, Brian J. Murrell wrote: > > Even if they end up in 2:3, they should at least be treated fairly. > > 2:3 will not be treated at all as long as 2:1 and 2:2 (which have higher > priority) are occupied.Right. I think I said in my last message that both 2:1 and 2:2 were very quiet (which is what I would expect) and that there was a lot of traffic going through 2:3. My point there was that even if 2:3 was processing a lot of packets, since it is an SFQ, everybody should eventually get use of it.> If the queues in 2:1 and 2:2 resp. never empty, > the packets in 2:3 will never be sent.Understood. But they were empty and lots of packets from 2:3 were being sent.> There is no fair treatment in PRIO.No, it''s priority based. Got that. Exactly what I am looking for in fact.> That''s the whole purpose of this scheduler, to give one band of packets > absolute priority over the other.Yup. My "interactive"/latency sensitive traffic should always get best service.> This is just out of personal interest, but could you try using instead of > your TBF qdisc, a very simple HTB Qdisc / class with the same bandwidth > limitation?Hrm. OK. But I''m not sure how I would use HTB to replace a classless TBF though. for TBF I use: tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1 So to replace that with HTB I tried: tc qdisc add dev ppp0 root handle 1: htb default 10 tc class add dev ppp0 parent 1 classid 1:1 htb rate 120kbit But nothing seems to be getting put into any of the child classes which are configured as: tc qdisc add dev ppp0 parent 1:1 handle 2: prio bands 3 tc qdisc add dev ppp0 parent 2:1 handle 10: sfq perturb 20 tc qdisc add dev ppp0 parent 2:2 handle 20: sfq perturb 20 tc qdisc add dev ppp0 parent 2:3 handle 30: sfq perturb 20 I''m probably misunderstanding all of the class naming and handles and such.> If that solves the problem, then you''re suffering from a > problem that I failed to solve when I last tried to use TBF; for some > reason it got stuck on me too.Well, TBF does not seem to be getting stuck. There is still lots of traffic moving when these other flows seem to just stop, so TBF can''t be the problem can it? It has to be PRIO, not dequeuing anything for these particular stalled flows to TBF right?> I don''t understand what you mean by fair treatment, but do try putting all > ACKs into high priority band, then it will have to be dequeued.I think I am doing that. I thought that is what: 859K 42M CLASSIFY tcp -- * ppp0 0.0.0.0/0 0.0.0.0/0 tcp flags:0x16/0x10 length 0:128 CLASSIFY set 2:1 in the POSTROUTING chain should be doing. It''s the result of the iptables rule: iptables -t mangle -I POSTROUTING -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST,ACK ACK -m length --length :128 -j CLASSIFY --set-class 2:1 b. -- My other computer is your Microsoft Windows server. Brian J. Murrell _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Brian J. Murrell wrote:> I thought I had this all worked out, but it seems not. The following tc > configuration: > > tc qdisc del dev ppp0 root 2> /dev/null > /dev/null > tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1> But it seems that some outbound flows are being blocked entirely. I > don''t think they are being starved though. Even if they end up in 2:3, > they should at least be treated fairly. But I am producing a flow to > 66.1.2.3 which does increment the counters in 2:1 but after a few > packets the flow stalls:Your burst is too small. It needs to be at least one MTU.
On Sunday 04 December 2005 19:17, Brian J. Murrell wrote:> > There is no fair treatment in PRIO. > > No, it''s priority based. Got that. Exactly what I am looking for in > fact.Sorry, seems that I misunderstood you in your message in the point that you meant SFQ and not PRIO when you were talking about fair treatment.> So to replace that with HTB I tried: > > tc qdisc add dev ppp0 root handle 1: htb default 10 > tc class add dev ppp0 parent 1 classid 1:1 htb rate 120kbitWithout additional filter rules, it should be ''default 1'', and not 10. Otherwise HTB will try to put the classes in 1:10, which does not exist, and instead send them out directly without any shaping at all.> Well, TBF does not seem to be getting stuck. There is still lots of > traffic moving when these other flows seem to just stop, so TBF can''t be > the problem can it? It has to be PRIO, not dequeuing anything for these > particular stalled flows to TBF right?Hmmm. Well, I was just guessing, because I had this ''stuck'' problem with TBF before. As I said, I never really solved that problem, so I can''t say much about cause and such. The way I understand it, the root qdisc will get the request to dequeue a packet, and it will forward this request to the underlying qdiscs. So when TBF is asked for a packet, it will decide wether to send one or not (depending on wether there is any bandwidth left). If it wants to dequeue one, it will forward this request to the underlying queue(s), PRIO in your case. PRIO will look at it''s bands and dequeue a packet from the first band that has packets queued; in your case, it will have to ask the underlying SFQ queue to select a packet. SFQ will look at the packets it has queued and select one based on it''s "stochastical fairness" algorithm. SFQ then returns this packet to PRIO which in turn will return this packet to TBF which in turn will send this packet on it''s way. Please note that this is all guesswork. I haven''t actually looked at the kernel code for all of that. :-) It might actually work in a totally different way. Anyway, if my understanding model is correct, and some packets in your prio bands get sent and others don''t, it should be the fault of SFQ, and not PRIO, for selecting the wrong packets. My guess before was that TBF was at fault, allowing too little bandwidth, which would lead to a general stalling, which would be most noticeable on connections that are bandwidth intensive. A completely different reason may be that you''ve got a bad mixing of flows; for example, "important" traffic like web browsing etc., and P2P don''t go well together in the same queue. This is simply because P2P has the habit of opening hundreds of connection, whereas WWW is just one or at least very few connections. So if you''ve got maybe 5 WWW connections and 200 P2P connections flowing through the same SFQ queue, every connection will be able to send about the same number of packets, resulting in a lot of P2P packets and very few WWW packets, just because there are so many more P2P connections there. For that very reason, in my P2P setups I''m actually using 4 prio bands, putting P2P alone into the 4th band, so that it may starve when there is any traffic other than P2P.> I think I am doing that. I thought that is what: > > iptables -t mangle -I POSTROUTING -o ppp0 -p tcp -m tcp --tcp-flags > SYN,RST,ACK ACK -m length --length :128 -j CLASSIFY --set-class 2:1I apologize; I''m guilty of not reading your messages carefully enough. Regards, Andreas Klauer
Patrick McHardy wrote:> Brian J. Murrell wrote: > >> I thought I had this all worked out, but it seems not. The following tc >> configuration: >> >> tc qdisc del dev ppp0 root 2> /dev/null > /dev/null >> tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1 > > >> But it seems that some outbound flows are being blocked entirely. I >> don''t think they are being starved though. Even if they end up in 2:3, >> they should at least be treated fairly. But I am producing a flow to >> 66.1.2.3 which does increment the counters in 2:1 but after a few >> packets the flow stalls: > > > Your burst is too small. It needs to be at least one MTU.Patrick do you know why prio is requeueing here? I see the same testing with (a fixed) Brian''s setup. If it''s to get length then it''s going to be a bit out sometimes with sfq attached - it will also messup sfq fairness whatever the reason. Brian - limit is in bytes though you get away with it here by adding prio/sfq, limit 1 would stop traffic if tbf was alone. Also if you ever try tbf on ethX then burst/mtu/limit need to be your mtu + 14. SFQ causes packet reordering when it perturbs and is best for bulk traffic. I would put a shortish bfifo on interactive class (though in practice you''ve kindof lost the battle if your interactive has queued, so its sfq should be empty when a packet arrives anyway). Andy.
On Sun, 2005-12-04 at 19:46 +0100, Patrick McHardy wrote:> > Your burst is too small. It needs to be at least one MTU.Bingo! I guess it''s obvious that I don''t really understand how burst works. :-) But setting it to 1600 seems to have solved the problem. Thanx! b. -- My other computer is your Microsoft Windows server. Brian J. Murrell _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc