Hi, I''m experimenting using the HTB queueing discipline for traffic shaping. However, it is not really exact. Currently, I try this setup: tc qdisc add dev eth0 root handle 1: htb default 3 tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB tc qdisc add dev eth0 parent 1:1 sfq tc class add dev eth0 parent 1: classid 1:2 htb rate 3Mbit burst 2kB tc qdisc add dev eth0 parent 1:2 sfq tc class add dev eth0 parent 1: classid 1:3 htb rate 77Mbit burst 150kB tc qdisc add dev eth0 parent 1:3 sfq tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 1 fw classid 1:1 tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 2 fw classid 1:2 /usr/local/sbin/iptables -t mangle -F OUTPUT /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9021 -j MARK --set-mark 1 /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9022:9023 -j MARK --set-mark 2 The network adaptor is connected to a 100MBit switch. When testing with netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec via class 1:1, both measured by one/multiple instances of netio and the rate output of "tc -s class dev eth0". This effect occurs with Linux 2.4.16, kernel either compiled with HZ set to 100 or 1024, and of course independant of the filter type used. During the tests, no packets need to be dropped, htb just delays. Output when two netio instances are sending to class 1:2 for some time: root@barkeeper /root/ >tc -s class list dev eth0 [...] class htb 1:2 root leaf 800e: prio 0 rate 3Mbit ceil 3Mbit burst 2Kb cburst 2Kb Sent 236734498 bytes 156441 pkts (dropped 0, overlimits 301823) rate 378286bps 250pps backlog 44p lended: 156397 borrowed: 0 giants: 0 injects: 0 tokens: -3714 ctokens: -3714 [...] tbf seems not to be able to do exact rate limiting, too. Anything wrong in my configuration, or am I just taking false statistics? Stefan
On Thu, Dec 06, 2001 at 08:28:07PM +0100, Stefan Rompf wrote:> Hi, > > I''m experimenting using the HTB queueing discipline for traffic shaping. > However, it is not really exact. Currently, I try this setup: > > tc qdisc add dev eth0 root handle 1: htb default 3 > > tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB > tc qdisc add dev eth0 parent 1:1 sfqRaise burst, for 20Mbit it needs to be at least 20kbyte!> The network adaptor is connected to a 100MBit switch. When testing with > netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec > via class 1:1, both measured by one/multiple instances of netio and the > rate output of "tc -s class dev eth0". This effect occurs with Linux > 2.4.16, kernel either compiled with HZ set to 100 or 1024, and ofThe rate output of tc -s class is not that reliable.> tbf seems not to be able to do exact rate limiting, too. Anything wrong > in my configuration, or am I just taking false statistics?Both I think. There are some indications that raising HZ may not work, as it may not raise the resolution of the timers in the Linux Kernel - I am investigating this! So try keeping HZ stable, and raising the size of your bucket. Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services Trilab The Technology People Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
On Thu, Dec 06, 2001 at 08:48:11PM +0100, bert hubert wrote:> > tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB > > tc qdisc add dev eth0 parent 1:1 sfq > > Raise burst, for 20Mbit it needs to be at least 20kbyte!Is there any good documentation on setting minburst/maxburst on CBQ as well as burst rates for other queuing disciplines as mentioned above? -- Michael T. Babcock CTO, FibreSpeed Ltd. (Hosting, Security, Consultation, Database, etc) http://www.fibrespeed.net/~mbabcock/
On Thu, Dec 06, 2001 at 02:59:40PM -0500, Michael T. Babcock wrote:> On Thu, Dec 06, 2001 at 08:48:11PM +0100, bert hubert wrote: > > > tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB > > > tc qdisc add dev eth0 parent 1:1 sfq > > > > Raise burst, for 20Mbit it needs to be at least 20kbyte! > > Is there any good documentation on setting minburst/maxburst on CBQ as > well as burst rates for other queuing disciplines as mentioned above?HTB is implemented on its own page, and mentions the bucket sizes needed. CBQ is only documented on http://ds9a.nl/lartc ... (it also mentions minburst). Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services Trilab The Technology People Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
On Thu, Dec 06, 2001 at 08:28:07PM +0100, Stefan Rompf wrote:> Hi, > > tc qdisc add dev eth0 root handle 1: htb default 3 > > tc class add dev eth0 parent 1: classid 1:1 htb rate 20Mbit burst 4kB > tc qdisc add dev eth0 parent 1:1 sfq > > tc class add dev eth0 parent 1: classid 1:2 htb rate 3Mbit burst 2kB > tc qdisc add dev eth0 parent 1:2 sfq > > tc class add dev eth0 parent 1: classid 1:3 htb rate 77Mbit burst 150kB > tc qdisc add dev eth0 parent 1:3 sfq > > tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 1 fw classid > 1:1 > tc filter add dev eth0 parent 1:0 prio 7 protocol ip handle 2 fw classid > 1:2 > > /usr/local/sbin/iptables -t mangle -F OUTPUT > /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9021 -j MARK > --set-mark 1 > /usr/local/sbin/iptables -t mangle -A OUTPUT -p tcp --dport 9022:9023 -j > MARK --set-mark 2 > > The network adaptor is connected to a 100MBit switch. When testing with > netio, I can send up to 370kB/sec through class 1:2 and up to 2,4MB/sec > via class 1:1, both measured by one/multiple instances of netio and the > rate output of "tc -s class dev eth0". This effect occurs with Linux > 2.4.16, kernel either compiled with HZ set to 100 or 1024, and of > course independant of the filter type used. During the tests, no packets > need to be dropped, htb just delays.I think your results are quite exact. The difference is because of different units. 370kB = 370 * 8 = 2.96 Mbit 2.4MB = 19.2 Mbit -- Daniel
Hi,> I think your results are quite exact. The difference is because of > different units. > 370kB = 370 * 8 = 2.96 Mbit > 2.4MB = 19.2 Mbitobviously ;-) When calculating, I did remember that one megabit is not 1000000 bits, but forgot about the byte. Thanks for bringing that into my mind again. Stefan
> obviously ;-) When calculating, I did remember that one megabit is not > 1000000 bits, but forgot about the byte. Thanks for bringing that into > my mind again.One megabit IS actually 10^6 bit - take a look at: http://physics.nist.gov/cuu/Units/binary.html What you are thinking about is one mebibit, which is 2^20 bit, take a look at the boot message of a recent kernel and see those prefix''es used for ide disk cache sice. - David
On Thu, Dec 06, 2001 at 08:48:11PM +0100, bert hubert wrote:> On Thu, Dec 06, 2001 at 08:28:07PM +0100, Stefan Rompf wrote:...> > tbf seems not to be able to do exact rate limiting, too. Anything wrong > > in my configuration, or am I just taking false statistics? > > Both I think. There are some indications that raising HZ may not work, as it > may not raise the resolution of the timers in the Linux Kernel - I am > investigating this! So try keeping HZ stable, and raising the size of your > bucket.would it make any difference if you do... --- linux/include/net/pkt_sched.h.bak Sun Nov 19 20:14:10 2000 +++ linux/include/net/pkt_sched.h Sun Nov 19 20:20:34 2000 @@ -5,7 +5,7 @@ #define PSCHED_JIFFIES 2 #define PSCHED_CPU 3 -#define PSCHED_CLOCK_SOURCE PSCHED_JIFFIES +#define PSCHED_CLOCK_SOURCE PSCHED_CPU #include <linux/config.h> #include <linux/pkt_sched.h> I didn''t see PSCHED_CLOCK_SOURCE in the 2.4routing docs... Can you tell more about PSCHED_CLOCK_SOURCE ? -- Safari - safari@iki.fi - PGP key 0x427E7914 - http://www.iki.fi/safari/ The UNIX Guru''s View of Sex: "unzip ; strip ; touch ; finger ; \ mount ; fsck ; more ; yes ; umount ; sleep"
> > Both I think. There are some indications that raising HZ may not work, as it > > may not raise the resolution of the timers in the Linux Kernel - I am > > investigating this! So try keeping HZ stable, and raising the size of your > > bucket. > > would it make any difference if you do... > > -#define PSCHED_CLOCK_SOURCE PSCHED_JIFFIES > +#define PSCHED_CLOCK_SOURCE PSCHED_CPUby the way if you have pentium with tsc the patch above is stronly recommended. It allows for sub nanosecond precision... devik
On Fri, Dec 07, 2001 at 06:57:39PM +0200, Sami Farin wrote:> Ahu wrote: > > Both I think. There are some indications that raising HZ may not work, as it > > may not raise the resolution of the timers in the Linux Kernel - I am > > investigating this! So try keeping HZ stable, and raising the size of your > > bucket. > > would it make any difference if you do...Ok - I''ve since investigated this. Shapers use timers internally to block themselves for set periods of time. These timers appear to have HZ resolution, but a comment in the very good Linux Device Drivers book http://www.xml.com/ldd/chapter/book/ appeared to indicate that raising HZ would not raise their resolution. I contacted Jonathan Corbet, one of the authors, and he checked and found that the book was incorrect: raising HZ *does* increase timer resolution.> -#define PSCHED_CLOCK_SOURCE PSCHED_JIFFIES > +#define PSCHED_CLOCK_SOURCE PSCHED_CPU > > I didn''t see PSCHED_CLOCK_SOURCE in the 2.4routing docs... > Can you tell more about PSCHED_CLOCK_SOURCE ?As far as I can see, this only increases the *resolution*, not *precision*. In other words, you can''t specify shorter delays this way but you can measure a delay that already happened far more precisely. For example, if you decide to delay for 1 jiffy, 10ms, you may in fact have waited 14ms. PSCHED_JIFFIES will not see this - 1.4==1 jiffy has passed. PSCHED_CPU *will* see it, and let the right amount of packets out. But I''m hoping that Martin Devera can check this for me, devik? Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services Trilab The Technology People Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
On Sat, Dec 08, 2001 at 09:51:22AM +0100, Martin Devera wrote:> > > Both I think. There are some indications that raising HZ may not work, as it > > > may not raise the resolution of the timers in the Linux Kernel - I am > > > investigating this! So try keeping HZ stable, and raising the size of your > > > bucket. > > > > would it make any difference if you do... > > > > -#define PSCHED_CLOCK_SOURCE PSCHED_JIFFIES > > +#define PSCHED_CLOCK_SOURCE PSCHED_CPU > > by the way if you have pentium with tsc the patch above > is stronly recommended. It allows for sub nanosecond > precision... > devikIs there a reason not to do this? In which case we should probably submit a patch that makes PSCHED_CPU the default if you have a TSC. Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services Trilab The Technology People Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
> > > -#define PSCHED_CLOCK_SOURCE PSCHED_JIFFIES > > > +#define PSCHED_CLOCK_SOURCE PSCHED_CPU > > > > by the way if you have pentium with tsc the patch above > > is stronly recommended. It allows for sub nanosecond > > precision... > > devik > > Is there a reason not to do this? In which case we should probably submit a > patch that makes PSCHED_CPU the default if you have a TSC.It would be nice - but when ANK does somethink in some way there is reason usualy. I''ve already been thinking about it as #ifdef CONFIG_X86_TSC #define PSCHED_CLOCK_SOURCE PSCHED_CPU .... I can found no reason why don''t use TSC. With highly loaded interface where eth interrupts are faster than timer ones it should make flow smoother. Without TSC all packets within single timer tick (10ms) are treated as havin the same arival time. CBQ''s clock integrator based on HW speed computations overcomes this and it might be reason why ANK left JIFFIES here. If you think that you have such power to convince people about it then go ahead. :) devik
Hi,
I am trying to understand the burst and cburst parameters in HTB. Can
somebody explain it. The docs arent very clear.
I am trying to throttle a 10Mbit link to a T1 link using HTB. I am using
the following:
------------------------------------------
DEVICE="dev eth1"
BANDWIDTH="bandwidth 10Mbit"
LIMITBW="1.536Mbit"
AVPKT="avpkt 1470"
# Root HTB qdisc 1:
$TC qdisc add $DEVICE root handle 1: htb
# This HTB class rate limits everyting to 1Mbit
$TC class add $DEVICE parent 1: classid 1:1 htb rate 1.536Mbps ceil
1.536Mbps burst 1k cburst 0.5k
------------------------------------------
Is this right ?
Also, can HTB take other qdiscs and classes ?
Regards,
Amit
--
The statement below is true.
The statement above is false.
^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^
Amit Kucheria
EECS Grad. Research Assistant
University of Kansas @ Lawrence
(R)+1-(785)-830 8521 ||| (O)+1-(785)-864 7774
____________________________________________________
On Monday 10 December 2001 01:27, you wrote:> Hi, > > I am trying to understand the burst and cburst parameters in HTB. Can > somebody explain it. The docs arent very clear. > > I am trying to throttle a 10Mbit link to a T1 link using HTB. I am using > the following: > > ------------------------------------------ > DEVICE="dev eth1" > BANDWIDTH="bandwidth 10Mbit" > LIMITBW="1.536Mbit" > AVPKT="avpkt 1470" > > # Root HTB qdisc 1: > $TC qdisc add $DEVICE root handle 1: htb > > # This HTB class rate limits everyting to 1Mbit > $TC class add $DEVICE parent 1: classid 1:1 htb rate 1.536Mbps ceil > 1.536Mbps burst 1k cburst 0.5k > ------------------------------------------ > Is this right ?I think so. Is not working ? ? Stef -- stef.coene@docum.org More QOS info : http://docum.org/ Title : "Using Linux as bandwidth manager"
> > I am trying to understand the burst and cburst parameters in HTB. Can > > somebody explain it. The docs arent very clear. > > > > I am trying to throttle a 10Mbit link to a T1 link using HTB. I am using > > the following: > > > > ------------------------------------------ > > DEVICE="dev eth1" > > BANDWIDTH="bandwidth 10Mbit" > > LIMITBW="1.536Mbit" > > AVPKT="avpkt 1470" > > > > # Root HTB qdisc 1: > > $TC qdisc add $DEVICE root handle 1: htb > > > > # This HTB class rate limits everyting to 1Mbit > > $TC class add $DEVICE parent 1: classid 1:1 htb rate 1.536Mbps ceil > > 1.536Mbps burst 1k cburst 0.5k > > ------------------------------------------ > > Is this right ? > I think so. > Is not working ? ?Not really. It kills all the traffic...giving me very low thruput....i read on the HTB homepage that on x86 machines burst has to be a minimum of 12K or something like that. Whats cburst ? -- The statement below is true. The statement above is false. ^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^ Amit Kucheria EECS Grad. Research Assistant University of Kansas @ Lawrence (R)+1-(785)-830 8521 ||| (O)+1-(785)-864 7774 ____________________________________________________
> Not really. > > It kills all the traffic...giving me very low thruput....i read on the HTB > homepage that on x86 machines burst has to be a minimum of 12K or > something like that. > > Whats cburst ?If you set cbust equal to packet size, the rate will be checked each time you send cbust bytes and thus each time you send a packet. This would be ideal and you will never get a bust that exceed the ceil. Default cburst = 2Kb/8. But how do you get a burst? That''s why you have the burst parameter. Each time you can poll the hardware, you can change to a new class. Devik calculated the burst for a i386 on a 10mbit and it''s 12kb. So, you can send 12kb, choose an other class, send an other one, choose an other class. That''s the fastest way you can work. The 12kb is sended as fast as possible and creates a burst. But with the cburst parameter you can control that burst. If you specify a bigger burst parameter, you can create bigger burst, but unload the CPU/hardware. I hope I''m explained it right. Devik ? Stef -- stef.coene@docum.org More QOS info : http://docum.org/ Title : "Using Linux as bandwidth manager"
> Not really. > > It kills all the traffic...giving me very low thruput....i read on the HTB > homepage that on x86 machines burst has to be a minimum of 12K or > something like that. > > Whats cburst ?If you set cbust equal to packet size, the rate will be checked each time you send cbust bytes and thus each time you send a packet. This would be ideal and you will never get a bust that exceed the ceil. Default cburst = 2Kb/8. But how do you get a burst? That''s why you have the burst parameter. Each time you can poll the hardware, you can change to a new class. Devik calculated the burst for a i386 on a 10mbit and it''s 12kb. So, you can send 12kb, choose an other class, send an other one, choose an other class. That''s the fastest way you can work. The 12kb is sended as fast as possible and creates a burst. But with the cburst parameter you can control that burst. If you specify a bigger burst parameter, you can create bigger burst, but unload the CPU/hardware. I hope I''m explained it right. Devik ? Stef -- stef.coene@docum.org More QOS info : http://docum.org/ Title : "Using Linux as bandwidth manager"
On Mon, 10 Dec 2001, Stef Coene wrote:> > Not really. > > > > It kills all the traffic...giving me very low thruput....i read on the HTB > > homepage that on x86 machines burst has to be a minimum of 12K or > > something like that. > > > > Whats cburst ? > If you set cbust equal to packet size, the rate will be checked each time you > send cbust bytes and thus each time you send a packet. This would be ideal > and you will never get a bust that exceed the ceil. Default cburst = 2Kb/8. > > But how do you get a burst? That''s why you have the burst parameter. Each > time you can poll the hardware, you can change to a new class. Devik > calculated the burst for a i386 on a 10mbit and it''s 12kb. So, you can send > 12kb, choose an other class, send an other one, choose an other class. > That''s the fastest way you can work. The 12kb is sended as fast as possible > and creates a burst. But with the cburst parameter you can control that > burst. If you specify a bigger burst parameter, you can create bigger burst, > but unload the CPU/hardware. > > I hope I''m explained it right. Devik ?Almost :) The part "send 12k and select other class" is not exact. Other class is selected also when current one is empty or DRR allotment is exceeded. Also cburst is not meant for burst reshaping (like peakrate in tbf) but can be used for it. tbf estimators always needs two parameters, rate and burst. There are two estimators (rate and ceil) thus we have four (2*2) parameters: rate/burst ceil/cburst I''m working on new patch to tc where you have not need to set bursts. There are automaticaly computed from rate and ceil to their smallest possible values (but you can still override them). devik
On Mon, 10 Dec 2001, Martin Devera wrote:> > I hope I''m explained it right. Devik ? > > Almost :) The part "send 12k and select other class" is not exact. > Other class is selected also when current one is empty or DRR allotment > is exceeded. > Also cburst is not meant for burst reshaping (like peakrate in tbf) > but can be used for it. > tbf estimators always needs two parameters, rate and burst. There are > two estimators (rate and ceil) thus we have four (2*2) parameters: > rate/burst > ceil/cburstHmm...so if i understand this correctly, ''burst'' controls ''rate'' and ''cburst'' controls ''ceil''? Lets say that i am generating bursty fixed length UDP traffic i.e.1470 bytes every 5000us. Now for a pkt size of 1470 bytes and ''rate'' and ''ceil'' being 1.536Mbit what would be the ideal values of burst and cburst? -- The statement below is true. The statement above is false. ^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^ Amit Kucheria EECS Grad. Research Assistant University of Kansas @ Lawrence (R)+1-(785)-830 8521 ||| (O)+1-(785)-864 7774 ____________________________________________________
> Hmm...so if i understand this correctly, ''burst'' controls ''rate'' and > ''cburst'' controls ''ceil''?exactly.> Lets say that i am generating bursty fixed length UDP traffic i.e.1470 > bytes every 5000us. > Now for a pkt size of 1470 bytes and ''rate'' and ''ceil'' being 1.536Mbit > what would be the ideal values of burst and cburst?So one 1.5k packet then 5ms pause, other 1.5 packet ... ok ? It leads to rate cca 1500/5ms = 300kBps = 2.4mbit. Because is exceeds your 1.5mbit rate so that the burst value is not so important. Just use minimal value: burstmin = 1.5mbit/8*10ms = 2k devik
On Tue, 11 Dec 2001, devik wrote:> > Hmm...so if i understand this correctly, ''burst'' controls ''rate'' and > > ''cburst'' controls ''ceil''? > > exactly. > > > Lets say that i am generating bursty fixed length UDP traffic i.e.1470 > > bytes every 5000us. > > Now for a pkt size of 1470 bytes and ''rate'' and ''ceil'' being 1.536Mbit > > what would be the ideal values of burst and cburst? > > So one 1.5k packet then 5ms pause, other 1.5 packet ... ok ? > It leads to rate cca 1500/5ms = 300kBps = 2.4mbit. > > Because is exceeds your 1.5mbit rate so that the burst value > is not so important. Just use minimal value: > > burstmin = 1.5mbit/8*10ms = 2k > > devikUsing those parameters puts the cap on my bandwidth real hard, not at all accuate. Infact, my connections go haywire. Here are the results. ----------------------------------------------------------------- testbed16 [30] % sudo ./T1-using-htb TC=/usr/bin/tc DEVICE=dev eth1 BANDWIDTH=bandwidth 10Mbit LIMITBW=1Mbit AVPKT=avpkt 1470 SRC=192.168.157.2 DEST=192.168.158.2 tc qdisc add dev eth1 root handle 1: htb tc class add dev eth1 parent 1: classid 1:1 htb rate 1Mbit ceil 1Mbit burst 2k ----------------------------------------------------------------- testbed16 [32] % showinfo eth1 ---------------------------------- QDISC logs ---------------------------------- qdisc htb 1: dev eth1 r2q 10 default 1 dcache 0 deq_util 1/1000000 deq_rate 0 trials_per_deq 0 dcache_hits 0 Sent 2946644 bytes 2105 pkts (dropped 1209, overlimits 9640) ---------------------------------- CLASS logs ---------------------------------- class htb 1:1 root prio 0 rate 1Mbit ceil 1Mbit burst 2Kb cburst 2Kb Sent 2946644 bytes 2105 pkts (dropped 1209, overlimits 8506) rate 283bps lended: 2105 borrowed: 0 giants: 0 injects: 0 tokens: 15491 ctokens: 15491 ----------------------------------------------------------------- When i increase the burst size to say 100k i get better thruput but still not equivalent to the ceil. Regards, Amit -- The statement below is true. The statement above is false. ^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^ Amit Kucheria EECS Grad. Research Assistant University of Kansas @ Lawrence (R)+1-(785)-830 8521 ||| (O)+1-(785)-864 7774 ____________________________________________________