Send LARTC mailing list submissions to
lartc@mailman.ds9a.nl
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.ds9a.nl/mailman/listinfo/lartc
or, via email, send a message with subject or body ''help'' to
lartc-request@mailman.ds9a.nl
You can reach the person managing the list at
lartc-admin@mailman.ds9a.nl
When replying, please edit your Subject line so it is more specific
than "Re: Contents of LARTC digest..."
Today''s Topics:
1. Re: CBQ and all other qdiscs now REALLY completely documented (jamal)
2. Re: more on cbq parameters (Michael T. Babcock)
3. Re: Re: further CBQ/tc documentation ds9a.nl/lartc/manpages (Michael T.
Babcock)
4. Re: Re: further CBQ/tc documentation ds9a.nl/lartc/manpages (Michael T.
Babcock)
5. quit (johannes.ebenhoeh@a1.net)
6. HTB burst, cburst parameters (Amit Kucheria)
7. CBQ MANPAGE: I hear the theme of ''2001, A Space Odyssey''
(bert hubert)
8. Re: CBQ MANPAGE: I hear the theme of ''2001, A Space
Odyssey'' (jamal)
--__--__--
Message: 1
Date: Sun, 9 Dec 2001 16:45:01 -0500 (EST)
From: jamal <hadi@cyberus.ca>
To: <kuznet@ms2.inr.ac.ru>
Cc: <ahu@ds9a.nl>, <lartc@mailman.ds9a.nl>,
<linux-kernel@vger.kernel.org>, <netdev@oss.sgi.com>
Subject: [LARTC] Re: CBQ and all other qdiscs now REALLY completely documented
On Sun, 9 Dec 2001 kuznet@ms2.inr.ac.ru wrote:
> Hello!
>
> > > But to do this, you would need to be able to set skb->priority
to a 32bit
> > > number:
> > >
> >
> > Cant think of a straight way to do this .... Alexey would know,
>
> SO_PRIORITY. Or I did not follow you?
>
So priority limits the size of skb->priority to be from 0..6; this wont
work with that check in cbq.
cheers,
jamal
--__--__--
Message: 2
Date: Sun, 9 Dec 2001 17:28:44 -0500
From: "Michael T. Babcock" <mbabcock@fibrespeed.net>
To: LARTC List <lartc@mailman.ds9a.nl>
Subject: Re: [LARTC] more on cbq parameters
On Sat, Dec 08, 2001 at 09:10:50PM +0100, bert hubert
wrote:> > Notice above I supplied bandwidth 30kbit which is far from the actual
> > physical bandwidth (100Mbit). Maybe this is why I get good results.
> > Maybe this is what you''re SUPPOSED to do!
>
> Not that I''m aware of.
To agree with you, AFAICS, the correct way to deal with this is to specify
the root bandwidth as the maximum physical bandwidth on the interface, then
split it down using classes that have rates set to the expected rates.
On a 100Mbit card connected to a 256kbit line, I used something like:
tc qdisc add dev eth0 root handle 1: cbq \
bandwidth 100Mbit avpkt 1000
tc class add dev eth0 parent 1:0 classid 1:1 cbq \
bandwidth 100Mbit rate 256kbit [...]
tc qdisc add dev eth0 parent 1:1 handle 10: cbq \
bandwidth 256kbit allot 1514 avpkt 1000
(PS, highly inspired by Stef and others'' scripts of course)
All my other classes then hang off 10: instead of 1: and work quite well.
What I''ve considered doing a few times is adding an option to dump out
the values CBQ is looking at for idleness at each level as well as
dynamic avpkt values (based on reality).
HTB may do this, of course.
--
Michael T. Babcock
CTO, FibreSpeed Ltd. (Hosting, Security, Consultation, Database, etc)
http://www.fibrespeed.net/~mbabcock/
--__--__--
Message: 3
Date: Sun, 9 Dec 2001 17:33:13 -0500
From: "Michael T. Babcock" <mbabcock@fibrespeed.net>
To: LARTC List <lartc@mailman.ds9a.nl>
Subject: Re: [LARTC] Re: further CBQ/tc documentation ds9a.nl/lartc/manpages
On Sat, Dec 08, 2001 at 03:43:05PM -0500, jamal wrote:> - You keep saying "reodering" -- dont know what that means.
Reordering is
> generally considered a Bad Thing(tm).
Reordering happens on a mass scale (packets often go out in a different order
than they were received / generated) but not on a per-qdisc scale (packets
go out ''in order'' within an SFQ queue or within a CBQ queue).
Its quite
obvious that fairness causes overall reordering of the available packets
because you sometimes with to pass along (for example) an SSH packet before
the 10 waiting FTP packets even though the latter got there first.
--
Michael T. Babcock
CTO, FibreSpeed Ltd. (Hosting, Security, Consultation, Database, etc)
http://www.fibrespeed.net/~mbabcock/
--__--__--
Message: 4
Date: Sun, 9 Dec 2001 17:36:52 -0500
From: "Michael T. Babcock" <mbabcock@fibrespeed.net>
To: LARTC List <lartc@mailman.ds9a.nl>
Subject: Re: [LARTC] Re: further CBQ/tc documentation ds9a.nl/lartc/manpages
On Sat, Dec 08, 2001 at 10:30:55PM +0100, bert hubert
wrote:> The ingress qdisc is a strange animal in that is not used to send packets
> out to the network adaptor. Instead, it allows you to apply tc filters to
> packets coming in over the interface, regardless of whether they have a
> local destination or are to be forwarded.
Opinions, opinions ... just the facts please. My suggested paragraph:
The ingress qdisc allows the application of tc filters to the inbound
packets on an interface instead of the outgoing ones. This filtering
is done to all incoming packets, whether destined for the local host or
to be forwarded.
--
Michael T. Babcock
CTO, FibreSpeed Ltd. (Hosting, Security, Consultation, Database, etc)
http://www.fibrespeed.net/~mbabcock/
--__--__--
Message: 5
From: johannes.ebenhoeh@a1.net <johannes.ebenhoeh@a1.net>
To: LARTC@mailman.ds9a.nl
Date: Mon, 10 Dec 2001 00:14:19 +0100
Subject: [LARTC] quit
--__--__--
Message: 6
Date: Sun, 9 Dec 2001 18:27:43 -0600 (CST)
From: Amit Kucheria <amitk@ittc.ku.edu>
To: <lartc@mailman.ds9a.nl>
Subject: [LARTC] HTB burst, cburst parameters
Hi,
I am trying to understand the burst and cburst parameters in HTB. Can
somebody explain it. The docs arent very clear.
I am trying to throttle a 10Mbit link to a T1 link using HTB. I am using
the following:
------------------------------------------
DEVICE="dev eth1"
BANDWIDTH="bandwidth 10Mbit"
LIMITBW="1.536Mbit"
AVPKT="avpkt 1470"
# Root HTB qdisc 1:
$TC qdisc add $DEVICE root handle 1: htb
# This HTB class rate limits everyting to 1Mbit
$TC class add $DEVICE parent 1: classid 1:1 htb rate 1.536Mbps ceil
1.536Mbps burst 1k cburst 0.5k
------------------------------------------
Is this right ?
Also, can HTB take other qdiscs and classes ?
Regards,
Amit
--
The statement below is true.
The statement above is false.
^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^
Amit Kucheria
EECS Grad. Research Assistant
University of Kansas @ Lawrence
(R)+1-(785)-830 8521 ||| (O)+1-(785)-864 7774
____________________________________________________
--__--__--
Message: 7
Date: Mon, 10 Dec 2001 01:41:30 +0100
From: bert hubert <ahu@ds9a.nl>
To: kuznet@ms2.inr.ac.ru
Cc: jamal <hadi@cyberus.ca>, lartc@mailman.ds9a.nl,
linux-kernel@vger.kernel.org, netdev@oss.sgi.com
Subject: [LARTC] CBQ MANPAGE: I hear the theme of ''2001, A Space
Odyssey''
--yrj/dFKFPuw6o+aM
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
... to the sound of ''Also sprach Zarathustra'':
After weeks of social deprivation and much digging through heaps of code, I
bring you
tc-cbq.8
The CBQ manpage. Nearly 2500 words, 8 printed pages, of nearly
unintelligible gobledygook, explaining mostly how CBQ works.
It is part of the Linux Advanced Routing & Traffic Control documentation
project which contains a HOWTO, a mailinglist, an IRC channel and now
manpages:
http://ds9a.nl/lartc
I want to thank Jamal for stubbornly straightening me out when I use messy
language and explaining how things work. The errors are mine though.
I *implore* ANK and others to read through this. I''m about exhausted
and
running out of time (need to get on with work), and have a hard time
figuring out the exact details of the CBQ link sharing algorithm. I need
help, so to speak. The manpage indicates where.
Thanks for your attention. Please find tc-cbq.8 attached.
Regards,
bert hubert
--
http://www.PowerDNS.com Versatile DNS Software & Services
Trilab The Technology People
Netherlabs BV / Rent-a-Nerd.nl - Nerd Available -
''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
--yrj/dFKFPuw6o+aM
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="tc-cbq.8"
.TH CBQ 8 "8 December 2001" "iproute2" "Linux"
.SH NAME
CBQ \- Class Based Queueing
.SH SYNOPSIS
.B tc qdisc ... dev
dev
.B ( parent
classid
.B | root) [ handle
major:
.B ] cbq avpkt
bytes
.B bandwidth
rate
.B [ cell
bytes
.B ] [ ewma
log
.B ] [ mpu
bytes
.B ]
.B tc class ... dev
dev
.B parent
major:[minor]
.B [ classid
major:minor
.B ] cbq allot
bytes
.B [ bandwidth
rate
.B ] [ rate
rate
.B ] prio
priority
.B [ weight
weight
.B ] [ minburst
packets
.B ] [ maxburst
packets
.B ] [ ewma
log
.B ] [ cell
bytes
.B ] avpkt
bytes
.B [ mpu
bytes
.B ] [ bounded isolated ] [ split
handle
.B & defmap
defmap
.B ] [ estimator
interval timeconstant
.B ]
.SH DESCRIPTION
Class Based Queueing is a classful qdisc that implements a rich
linksharing hierarchy of classes. It contains shaping elements as
well as prioritizing capabilities. Shaping is performed using link
idle time calculations based on the timing of dequeue events and
underlying link bandwidth.
.SH SHAPING ALGORITHM
Shaping is done using link idle time calculations, and actions taken if
these calculations deviate from set limits.
When shape a 10mbit/s connection to 1mbit/s, the link will
be idle 90% of the time. If it isn''t, it needs to be throttled so that
it
IS idle 90% of the time.
From the kernel''s perspecive, this is hard to measure, so CBQ instead
derives the idle time from the number of microseconds that elapse between
requests from the hardware layer for more data. Combined with the
knowledge of packet sizes, this is used to approximate how full or empty
the link is.
This is rather circumspect and doesn''t always arrive at proper
results. For example, what is the actual link speed of an interface
that is not really able to transmit the full 100mbit/s of data,
perhaps because of a badly implemented driver? A PCMCIA network card
will also never achieve 100mbit/s because of the way the bus is
designed - again, how do we calculate the idle time?
The physical link bandwidth may be ill defined in case of not-quite-real
network devices like PPP over Ethernet or PPTP over TCP/IP. The effective
bandwidth in that case is probably determined by the efficiency of pipes
to userspace - which not defined.
During operations, the effective idletime is measured using an
exponential weighted moving average (EWMA), which considers recent
packets to be exponentially more important than past ones. The unix
loadaverage is calculated in the same way.
The calculated idle time is substracted from the EWMA measured one,
the resulting number is called ''avgidle''. A perfectly loaded
link has
an avgidle of zero: packets arrive exactly at the calculated
interval.
An overloaded link has a negative avgidle and if it gets too negative,
CBQ throttles and is then ''overlimit''.
Conversely, an idle link might amass a huge avgidle, which would then
allow infinite bandwidths after a few hours of silence. To prevent
this, avgidle is capped at
.B maxidle.
If overlimit, in theory, the CBQ could throttle itself for exactly the
amount of time that was calculated to pass between packets, and then
pass one packet, and throttle again. Due to timer resolution constraints,
this may not be feasible, see the
.B minburst
parameter below.
.SH CLASSIFICATION
Within the one CBQ instance many classes may exist. Each of these classes
contains another qdisc, by default
.BR tc-pfifo (8).
When enqueueing a packet, CBQ starts at the root and uses various methods to
determine which class should receive the data. If a verdict is reached, this
process is repeated for the recipient class which might have further
means of classifying traffic to its children, if any.
CBQ has the following methods available to classify a packet to any child
classes.
.TP
(i)
.B skb->priority class encoding.
Can be set from userspace by an application with the
.B IP_PRIO
setsockopt.
The
.B skb->priority class encoding
only applies if the skb->priority holds a major:minor handle of an existing
class within this qdisc.
.TP
(ii)
tc filters attached to the class.
.TP
(iii)
The defmap of a class, as set with the
.B split & defmap
parameters. The defmap may contain instructions for each possible Linux packet
priority.
.P
Each class also has a
.B level.
Leaf nodes, attached to the bottom of the class hierarchy, have a level of 0.
.SH CLASSIFICATION ALGORITHM
Classification is a loop, which terminates when a leaf class is found. At any
point the loop may jump to the fallback algorithm.
The loop consists of the following steps:
.TP
(i)
If the packet is generated locally and has a valid classid encoded within its
.B skb->priority,
choose it and terminate.
.TP
(ii)
Consult the tc filters, if any, attached to this child. If these return
a class which is not a leaf class, restart loop from the class returned.
If it is a leaf, choose it and terminate.
.TP
(iii)
If the tc filters did not return a class, but did return a classid,
try to find a class with that id within this qdisc.
Check if the found class is of a lower
.B level
than the current class. If so, and the returned class is not a leaf node,
restart the loop at the found class. If it is a leaf node, terminate.
If we found an upward reference to a higher level, enter the fallback
algorithm.
.TP
(iv)
If the tc filters did not return a class, nor a valid reference to one,
consider the minor number of the reference to be the priority. Retrieve
a class from the defmap of this class for the priority. If this did not
contain a class, consult the defmap of this class for the
.B BEST_EFFORT
class. If this is an upward reference, or no
.B BEST_EFFORT
class was defined,
enter the fallback algorithm. If a valid class was found, and it is not a
leaf node, restart the loop at this class. If it is a leaf, choose it and
terminate. If
neither the priority distilled from the classid, nor the
.B BEST_EFFORT
priority yielded a class, enter the fallback algorithm.
.P
The fallback algorithm resides outside of the loop and is as follows.
.TP
(i)
Consult the defmap of the class at which the jump to fallback occured. If
the defmap contains a class for the
.B
priority
of the class (which is related to the TOS field), choose this class and
terminate.
.TP
(ii)
Consult the map for a class for the
.B BEST_EFFORT
priority. If found, choose it, and terminate.
.TP
(iii)
Choose the class at which breakout to the fallback algorithm occured. Terminate.
.P
The packet is enqueued to the class which was chosen when either algorithm
terminated. It is therefore possible for a packet to be enqueued *not* at a
leaf node, but in the middle of the hierarchy.
.SH LINK SHARING ALGORITHM
When dequeueing for sending to the network device, CBQ decides which of its
classes will be allowed to send. It does so with a Weighted Round Robin process
in which each class with packets gets a chance to send in turn. The WRR process
starts by asking the highest priority classes (lowest numerically -
highest semantically) for packets, and will continue to do so until they
have no more data to offer, in which case the process repeats for lower
priorities.
.B CERTAINTY ENDS HERE, ANK PLEASE HELP
Each class is not allowed to send at length though - they can only dequeue a
configurable amount of data during each round.
If a class is about to go overlimit, and it is not
.B bounded
it will try to borrow avgidle from siblings that are not
.B isolated.
This process is repeated from the bottom upwards. If a class is unable
to borrow enough avgidle to send a packet, it is throttled and not asked
for a packet for enough time for the avgidle to increase above zero.
.B I REALLY NEED HELP FIGURING THIS OUT. REST OF DOCUMENT IS PRETTY CERTAIN
.B AGAIN.
.SH QDISC
The root qdisc of a CBQ class tree has the following parameters:
.TP
parent major:minor | root
This mandatory parameter determines the place of the CBQ instance, either at the
.B root
of an interface or within an existing class.
.TP
handle major:
Like all other qdiscs, the CBQ can be assigned a handle. Should consist only
of a major number, followed by a colon. Optional.
.TP
avpkt bytes
For calculations, the average packet size must be known. It is silently capped
at a minimum of 2/3 of the interface MTU. Mandatory.
.TP
bandwidth rate
To determine the idle time, CBQ must know the bandwidth of your underlying
physical interface, or parent qdisc. This is a vital parameter, more about it
later. Mandatory.
.TP
cell
The cell size determines he granularity of packet transmission time
calculations. Has a sensible default.
.TP
mpu
A zero sized packet may still take time to transmit. This value is the lower
cap for packet transmission time calculations - packets smaller than this value
are still deemed to have this size. Defaults to zero.
.TP
ewma log
When CBQ needs to measure the average idle time, it does so using an
Exponentially Weighted Moving Average which smoothes out measurements into
a moving average. The EWMA LOG determines how much smoothing occurs. Defaults
to 5. Lower values imply greater sensitivity. Must be between 0 and 31.
.P
A CBQ qdisc does not shape out of its own accord. It only needs to know certain
parameters about the underlying link. Actual shaping is done in classes.
.SH CLASSES
Classes have a host of parameters to configure their operation.
.TP
parent major:minor
Place of this class within the hierarchy. If attached directly to a qdisc
and not to another class, minor can be omitted. Mandatory.
.TP
classid major:minor
Like qdiscs, classes can be named. The major number must be equal to the
major number of the qdisc to which it belongs. Optional, but needed if this
class is going to have children.
.TP
weight weight
When dequeueing to the interface, classes are tried for traffic in a
round-robin fashion. Classes with a higher configured qdisc will generally
have more traffic to offer during each round, so it makes sense to allow
it to dequeue more traffic. All weights under a class are normalized, so
only the ratios matter. Defaults to the configured rate, unless the priority
of this class is maximal, in which case it is set to 1.
.TP
allot bytes
Allot specifies how many bytes a qdisc can dequeue
during each round of the process. This parameter is weighted using the
renormalized class weight described above.
.TP
priority priority
In the round-robin process, classes with the lowest priority field are tried
for packets first. Mandatory.
.TP
rate rate
Maximum rate this class and all its children combined can send at. Mandatory.
.TP
bandwidth rate
This is different from the bandwidth specified when creating a CBQ disc. Only
used to determine maxidle and offtime, which are only calculated when
specifying maxburst or minburst. Mandatory if specifying maxburst or minburst.
.TP
maxburst
This number of packets is used to calculate maxidle so that when
avgidle is at maxidle, this number of average packets can be burst
before avgidle drops to 0. Set it higher to be more tolerant of
bursts. You can''t set maxidle directly, only via this parameter.
.TP
minburst
As mentioned before, CBQ needs to throttle in case of
overlimit. The ideal solution is to do so for exactly the calculated
idle time, and pass 1 packet. However, Unix kernels generally have a
hard time scheduling events shorter than 10ms, so it is better to
throttle for a longer period, and then pass minburst packets in one
go, and then sleep minburst times longer.
The time to wait is called the offtime. Higher values of minburst lead
to more accurate shaping in the long term, but to bigger bursts at
millisecond timescales.
.TP
minidle
If avgidle is below 0, we are overlimits and need to wait until
avgidle will be big enough to send one packet. To prevent a sudden
burst from shutting down the link for a prolonged period of time,
avgidle is reset to minidle if it gets too low.
Minidle is specified in negative microseconds, so 10 means that
avgidle is capped at -10us.
.TP
bounded
Signifies that this class will not borrow bandwidth from its siblings.
.TP
isolated
Means that this class will not borrow bandwidth to its siblings
.TP
split major:minor & defmap bitmap[/bitmap]
If consulting filters attached to a class did not give a verdict,
CBQ can also classify based on the packet''s priority. There are 16
priorities available, numbered from 0 to 15.
The defmap specifies which priorities this class wants to receive,
specified as a bitmap. The Least Significant Bit corresponds to priority
zero. The
.B split
parameter tells CBQ at which class the decision must be made, which should
be a (grand)parent of the class you are adding.
As an example, ''tc class add ... classid 10:1 cbq .. split 10:0 defmap
c0''
configures class 10:0 to send packets with priorities 6 and 7 to 10:1.
The complimentary configuration would then
be: ''tc class add ... classid 10:2 cbq ... split 10:0 defmap
3f''
Which would send all packets 0, 1, 2, 3, 4 and 5 to 10:1.
.TP
estimator interval timeconstant
CBQ can measure how much bandwidth each class is using, which tc filters
can use to classify packets with. In order to determine the bandwidth
it uses a very simple estimator that measures once every
.B interval
microseconds how much traffic has passed. This again is a EWMA, for which
the time constant can be specified, also in microseconds. The
.B time constant
corresponds to the sluggishness of the measurement or, conversely, to the
sensitivity of the average to short bursts. Higher values mean less
sensitivity.
.SH SOURCES
.TP
o
Sally Floyd and Van Jacobson, "Link-sharing and Resource
Management Models for Packet Networks",
IEEE/ACM Transactions on Networking, Vol.3, No.4, 1995
.TP
o
Sally Floyd, "Notes on CBQ and Guaranted Service", 1995
.TP
o
Sally Floyd, "Notes on Class-Based Queueing: Setting
Parameters", 1996
.TP
o
Sally Floyd and Michael Speer, "Experimental Results
for Class-Based Queueing", 1998, not published.
.SH SEE ALSO
.BR tc (8)
.SH AUTHOR
Alexey N. Kuznetsov, <kuznet@ms2.inr.ac.ru>. This manpage maintained by
bert hubert <ahu@ds9a.nl>
--yrj/dFKFPuw6o+aM--
--__--__--
Message: 8
Date: Sun, 9 Dec 2001 20:04:42 -0500 (EST)
From: jamal <hadi@cyberus.ca>
To: bert hubert <ahu@ds9a.nl>
Cc: <kuznet@ms2.inr.ac.ru>, <lartc@mailman.ds9a.nl>,
<linux-kernel@vger.kernel.org>, <netdev@oss.sgi.com>
Subject: [LARTC] Re: CBQ MANPAGE: I hear the theme of ''2001, A Space
Odyssey''
Sorry didnt read it; did the 30 sec scan ..
If this is meant to be for users, why are you talking about skb->priority?
Isnt it sufficient to just call it prioirity?
Also, if you think that Alexeys imp. is based on Floyd only, you are
highly mistaken;
Going back to high latency response mode ...
cheers,
jamal
On Mon, 10 Dec 2001, bert hubert wrote:
> ... to the sound of ''Also sprach Zarathustra'':
>
> After weeks of social deprivation and much digging through heaps of code,
I
> bring you
>
> tc-cbq.8
>
> The CBQ manpage. Nearly 2500 words, 8 printed pages, of nearly
> unintelligible gobledygook, explaining mostly how CBQ works.
>
> It is part of the Linux Advanced Routing & Traffic Control
documentation
> project which contains a HOWTO, a mailinglist, an IRC channel and now
> manpages:
>
> http://ds9a.nl/lartc
>
> I want to thank Jamal for stubbornly straightening me out when I use messy
> language and explaining how things work. The errors are mine though.
>
> I *implore* ANK and others to read through this. I''m about
exhausted and
> running out of time (need to get on with work), and have a hard time
> figuring out the exact details of the CBQ link sharing algorithm. I need
> help, so to speak. The manpage indicates where.
>
> Thanks for your attention. Please find tc-cbq.8 attached.
>
> Regards,
>
> bert hubert
>
>
> --
> http://www.PowerDNS.com Versatile DNS Software & Services
> Trilab The Technology People
> Netherlabs BV / Rent-a-Nerd.nl - Nerd Available -
> ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the
internet
>
--__--__--
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc
End of LARTC Digest