LARTC - Dec 2003 - Split bandwidth equally per IP

It''s me again, with the same problem.

I want to be able to split the bandwidth per IP, as I have 10 clients in my
LAN and I want to be sure they get the same rate when downloading.

 

I followed all the steps from the HTB manual, but the script can be
"beaten"
when one uses more connection for downloading ("FlashGet" or
"Download
Accelerator").

I have the same rates for all the 10 IP-s in my LAN.

 

Stef Coene told me to decrease the burst. (Indeed, 80k was a huge burst) I
did that, but it seams nothing''s changed.

Jon Zeeff said that sfq generally creates a queue per connection. (And that
is the reason why clients with more connections get more bandwidth)

 

So I need to change this "per connection" into "per IP".

Everything is clear, but I do not know what to change into my script to do
this.

I guess that I have to change "sfq perturb 10" into something else,
but I do
not know into what :)

 

 

Here is my script:

 

 

$TC qdisc add dev $LAN_IFACE root handle 1: htb default 30 

$TC class add dev $LAN_IFACE parent 1: classid 1:1 htb rate 120kbit ceil
120kbit burst 2k

 

 

#--------default class---------

$TC class add dev $LAN_IFACE parent 1:1 classid 1:30 htb rate 1kbit ceil
120kbit burst 2k 

$TC qdisc add dev $LAN_IFACE parent 1:30 handle 300: sfq perturb 10

 

 

 

 

#-----10 client classes-------------

 

$TC class add dev $LAN_IFACE parent 1:1 classid 1:11 htb rate 12kbit ceil
120kbit burst 2k 

$FILTER_IP 192.168.0.121 flowid 1:11 

$TC qdisc add dev $LAN_IFACE parent 1:11 handle 110: sfq perturb 10

 

$TC class add dev $LAN_IFACE parent 1:1 classid 1:12 htb rate 12kbit ceil
120kbit burst 2k 

$FILTER_IP 192.168.0.122 flowid 1:12 

$TC qdisc add dev $LAN_IFACE parent 1:12 handle 120: sfq perturb 10

 

 

etc...

 

 

Thanks again for your patience.

On Thursday 04 December 2003 19:52, Mihai Vlad wrote:
> It''s me again, with the same problem.
>
> I want to be able to split the bandwidth per IP, as I have 10 clients in my
> LAN and I want to be sure they get the same rate when downloading.
>
> I followed all the steps from the HTB manual, but the script can be
> "beaten" when one uses more connection for downloading
("FlashGet" or
> "Download Accelerator").
>
> I have the same rates for all the 10 IP-s in my LAN.
>
> Stef Coene told me to decrease the burst. (Indeed, 80k was a huge burst) I
> did that, but it seams nothing''s changed.
The burst parameter is not made for hugh bursts.  If you don''t specify
a
burst, htb will calculate the smallest one suitable for that class.
> Jon Zeeff said that sfq generally creates a queue per connection. (And that
> is the reason why clients with more connections get more bandwidth)
Yes and no.  Sfq creates 1 queue for a connection, but htb should distribute 
the bandwidth eaqual between the classes even if 1 class has more connections 
in it.
> So I need to change this "per connection" into "per
IP".
>
> Everything is clear, but I do not know what to change into my script to do
> this.
Can you post the output of
tc -s -d class show dev eth0 

Stef

-- 
stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.openprojects.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

I left out the burst parameter so that HTB would compute it.

The 80k burst parameter was my approach to prioritze somehow page browsing
against heavy downloads. I said that 80 kilos is the average size of a html
page.

Is there any other way to accomplish this task. Is there a way to tell page
browing from downloads?

Attached are my script and the output as they are quite long.

Thank you for your time, Stef.

On Thursday 04 December 2003 21:44, Mihai Vlad wrote:
> I left out the burst parameter so that HTB would compute it.
>
> The 80k burst parameter was my approach to prioritze somehow page browsing
> against heavy downloads. I said that 80 kilos is the average size of a html
> page.
>
> Is there any other way to accomplish this task. Is there a way to tell page
> browing from downloads?
You can use a burst 80kbit.  But.  If you use that burst, you need some time 
to regain the burst I again.  By that I mean the class has to stop sending 
packets so you can refill the token bucket.
So if you have a 10 rate and a 80 burst and you use all that burst, you have 
to stop sending packets for 8 seconds to refill the bucket and to reuse the 
burst.
> Attached are my script and the output as they are quite long.
I was interested in the output to see if the filters are working.  And they do 
because some classs have send packets.

Also try to add quantum 1500 or higher (10000) or so to the leaf classes.  A 
leaf class is the last class in the tree so it has no child classes.  The 
default quantum is based on the rate and that''s too low.

Stef

-- 
stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.openprojects.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Here are my untested (but simple) changes to SFQ to make it "fair" on
a per
ip basis.  I believe that this is actually what most people want - equal
bandwidth among a number of users without being tricked by users using many
connections.


In net/sched/sch_sfq.c:

                h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol == IPPROTO_TCP ||
                     iph->protocol == IPPROTO_UDP ||
                     iph->protocol == IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

change to:

h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol == IPPROTO_TCP ||
                     iph->protocol == IPPROTO_UDP ||
                     iph->protocol == IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Here are my untested (but simple) changes to SFQ to make it share 
outgoing bandwidth "fairly" per ip address (roughly, per local user)
instead of
being susceptible to being tricked by users with many connections. 
Don''t use
this on the wrong side of a NAT box where there is only one source ip address 
in use.

In net/sched/sch_sfq.c:

Change:

                h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol == IPPROTO_TCP ||
                     iph->protocol == IPPROTO_UDP ||
                     iph->protocol == IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

To:
                h = h2 = iph->saddr;
                break;


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Thanks very much for this reply.

In fact letting HTB calculate itself the burst parameter, and hard-coding
the quantum parameter for each leaf class (as Stef suggested), made the
traffic shaping much more accurate than it was before. (especially the
quantum settings > 1500).

I will make the changes in sch_sfq.c and keeep you all informed with the
results.



-----Original Message-----
From: jzeeff@internet2.edu [mailto:jzeeff@internet2.edu] 
Sent: Friday, December 05, 2003 5:19 PM
To: Stef Coene
Cc: Mihai Vlad; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Split bandwidth equally per IP


Here are my untested (but simple) changes to SFQ to make it share 
outgoing bandwidth "fairly" per ip address (roughly, per local user)
instead
of
being susceptible to being tricked by users with many connections. 
Don''t
use
this on the wrong side of a NAT box where there is only one source ip
address 
in use.

In net/sched/sch_sfq.c:

Change:

                h = iph->daddr;
                h2 = iph->saddr^iph->protocol;
                if (!(iph->frag_off&htons(IP_MF|IP_OFFSET)) &&
                    (iph->protocol == IPPROTO_TCP ||
                     iph->protocol == IPPROTO_UDP ||
                     iph->protocol == IPPROTO_ESP))
                        h2 ^= *(((u32*)iph) + iph->ihl);
                break;

To:
                h = h2 = iph->saddr;
                break;




_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

 : In fact letting HTB calculate itself the burst parameter, and
 : hard-coding the quantum parameter for each leaf class (as Stef
 : suggested), made the traffic shaping much more accurate than it was
 : before. (especially the quantum settings > 1500).

That Stef guy!  Always making good suggestions.

 : I will make the changes in sch_sfq.c and keeep you all informed with
 : the results.

Achtung!  There is already an esfq qdisc [0] which does this!  This patch
may be a good one, but since esfq already exists, perhaps you could try
that instead.

-Martin

 [0] http://www.ssi.bg/~alex/esfq/index.html

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Good point, what has to be done to make this part of the standard linux
kernel distribution?

> 
> Achtung!  There is already an esfq qdisc [0] which does this!  This patch
> may be a good one, but since esfq already exists, perhaps you could try
> that instead.
> 
> -Martin
> 
>  [0] http://www.ssi.bg/~alex/esfq/index.html
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello Martin, 

I have recently read your howto from:
http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/Traffi
c-Control-HOWTO.html

I quote: 

"Of the many types of contention for network bandwidth, this is one of the
easier types of contention to address in general. By using the SFQ qdisc,
traffic in a particular queue can be separated into flows, each of which
will be serviced fairly (inside that queue). Well-behaved applications (and
users) will find that using SFQ and ESFQ are sufficient for most sharing
needs. 

The Achilles heel of these fair queuing algorithms is a misbehaving user or
application which opens many connections simultaneously (e.g., eMule,
eDonkey, Kazaa). By creating a large number of individual flows, the
application can dominate slots in the fair queuing algorithm. Restated, the
fair queuing algorithm has no idea that a single application is generating
the majority of the flows, and cannot penalize the user. Other methods are
called for."







Can you post a real script using esfq, that splits the bandwidth equally per
IP? 

The documentation on esfq is scarce and I have no idea where to start from.

Thanks again for your time.








-----Original Message-----
From: Martin A. Brown [mailto:mabrown-lartc@securepipe.com] 
Sent: Friday, December 05, 2003 7:03 PM
To: Mihai Vlad
Cc: lartc@mailman.ds9a.nl
Subject: RE: [LARTC] Split bandwidth equally per IP

Achtung!  There is already an esfq qdisc [0] which does this!  This patch
may be a good one, but since esfq already exists, perhaps you could try
that instead.

-Martin

 [0] http://www.ssi.bg/~alex/esfq/index.html

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

ESFQ allows this to be "per ip address", which solves the problem.

I have not seen applications that grab many ip addresses for a single host,
although this is possible.

Quoting Mihai Vlad <mihaivlad@web-profile.net>:
> Hello Martin, 
> 
> I have recently read your howto from:
>
http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/Traffi
> c-Control-HOWTO.html
> 
> I quote: 
> 
> "Of the many types of contention for network bandwidth, this is one of
the
> easier types of contention to address in general. By using the SFQ qdisc,
> traffic in a particular queue can be separated into flows, each of which
> will be serviced fairly (inside that queue). Well-behaved applications (and
> users) will find that using SFQ and ESFQ are sufficient for most sharing
> needs. 
> 
> The Achilles heel of these fair queuing algorithms is a misbehaving user or
> application which opens many connections simultaneously (e.g., eMule,
> eDonkey, Kazaa). By creating a large number of individual flows, the
> application can dominate slots in the fair queuing algorithm. Restated, the
> fair queuing algorithm has no idea that a single application is generating
> the majority of the flows, and cannot penalize the user. Other methods are
> called for."
> 
> 
> 
> 
> 
> 
> 
> Can you post a real script using esfq, that splits the bandwidth equally
> per
> IP? 
> 
> The documentation on esfq is scarce and I have no idea where to start from.
> 
> Thanks again for your time.
> 
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Martin A. Brown [mailto:mabrown-lartc@securepipe.com] 
> Sent: Friday, December 05, 2003 7:03 PM
> To: Mihai Vlad
> Cc: lartc@mailman.ds9a.nl
> Subject: RE: [LARTC] Split bandwidth equally per IP
> 
> Achtung!  There is already an esfq qdisc [0] which does this!  This patch
> may be a good one, but since esfq already exists, perhaps you could try
> that instead.
> 
> -Martin
> 
>  [0] http://www.ssi.bg/~alex/esfq/index.html
> 
> -- 
> Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com
> 
> 
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 
> 


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello Mihai,

 : Can you post a real script using esfq, that splits the bandwidth
 : equally per IP?

I wish I had an example script to lend you....

To my recollection your issue was an obnoxious piece of software which was
generating many outbound flows.  This sort of a technique should prevent
any single IP from dominating your bandwidth (without your allowing that
domination):

  tc qdisc add dev $LAN_DEV parent $HTB_CLASS handle $ESFQ_HANDLE \
    esfq perturb 10 depth $UNIQUE_CLIENTS hash dst

This attaches an esfq qdisc to the class which contains competing
workstations/users for download bandwidth (in your environment).
Reminder!  We have already deNATted (is that a word) the inbound packets,
so the destination is the real workstation IP, and the

Here''s the help string [from the README]....

  Usage: ... esfq [ perturb SECS ] [ quantum BYTES ] [ depth FLOWS ]
  [ divisor HASHBITS ] [ limit PKTS ] [ hash HASHTYPE]

  Where:
  HASHTYPE := { classic | src | dst }

I''d agree that the easily available documentation for ESFQ is
minimal....I
would love to remedy that, so let me know how your experimentation with
the qdisc goes.  In your case, I''d suggest starting with a "hash
dst".

When I cautioned about Jon Zeeff''s patch [0], I didn''t
necessarily mean to
scare you away from it.  It may be a very good patch--I simply wanted to
point out that there already exists a tool to solve the problem you
appeared to need to solve.  Note also the recent list activity (by
Chijioke Kalu) with regard to kernel-2.6 and ESFQ [1].

Also, the section of my text (from tldp.org) you quoted was specifically
about the very same download accelerators which were giving you fits.
These tools simultaneously create a large number of TCP connections for a
single application-layer function (usually fetching a large file) in order
to (attempt to) defeat per-connection byte-rate limits.  The design of SFQ
did not anticipate this deliberate flooding attempt.  ESFQ takes a step
toward correcting this.

Again, I don''t know if ESFQ will solve your problem--I hope it does,
and I
hope to hear a report from you that this was the tool you needed.

Best of luck,

-Martin

  [0] http://mailman.ds9a.nl/pipermail/lartc/2003q4/010919.html
  [1] http://mailman.ds9a.nl/pipermail/lartc/2003q4/010939.html

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello all (again),

 : ESFQ allows this to be "per ip address", which solves the problem.
 :
 : I have not seen applications that grab many ip addresses for a single
 : host, although this is possible.

Would WRR be a better candidate for Mihai''s problem?  I have no
experience
with WRR.

Others?

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Thanks very much for your help.

It seems that Jim diGriz already accomplished that, but I did not read the
entire page to see that there was a script to download. My mistake :)

For those who don''t know yet, here is the link:
http://www.digriz.org.uk/jdg-qos-script/index.htm

I will try to do some testing with it, and I hope it will work. The only
thing I an afraid of is the "tiny" rates I have to use for each client
(7-8
Kbit).

I will keep you informed with my progress.




_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

I have used rates of 4 - 5kbits per system and works fine.  Also improved 
surfing by doing a TC+Squid patch.

But lets see how your''s goes.

K
>
>Thanks very much for your help.
>
>It seems that Jim diGriz already accomplished that, but I did not read the
>entire page to see that there was a script to download. My mistake :)
>
>For those who don''t know yet, here is the link:
>http://www.digriz.org.uk/jdg-qos-script/index.htm
>
>I will try to do some testing with it, and I hope it will work. The only
>thing I an afraid of is the "tiny" rates I have to use for each
client (7-8
>Kbit).
>
>I will keep you informed with my progress.
>
>
>
>
>_______________________________________________
>LARTC mailing list / LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_________________________________________________________________
Cell phone ‘switch’ rules are taking effect — find out more here. 
http://special.msn.com/msnbc/consumeradvocate.armx

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello, Martin, list,

Probably WRR would be a nice solution, indeed. But the question was is there
anyone on this list that could post a working (part of) script, that shares
bandwidth equally by source/destination hashes using ESFQ.

I''ve posted once or twice here on this subject, but of no avail. Maybe
half an
year ago I tried to find a working solution myself, but the bandwidth division
was not fair, at least not AS fair as I''d like. It is possible the
reason for
this to be the quite tiny amounts of bandwidth I gave to each connection (around
6 Kbits/s), but still...

I dont know, probably Alexander Atanasov (the author of this qdisc) could
enlighen us.


On Sun, 7 Dec 2003 00:23:18 -0600 (CST)
"''Martin A. Brown''"
<mabrown-lartc@securepipe.com> wrote:

 : Hello all (again),
 : 
 :  : ESFQ allows this to be "per ip address", which solves the
problem.
 :  :
 :  : I have not seen applications that grab many ip addresses for a single
 :  : host, although this is possible.
 : 
 : Would WRR be a better candidate for Mihai''s problem?  I have no
experience
 : with WRR.
 : 
 : Others?
 : 
 : -Martin

________________________________________________________________________

Cheers,
Nickola

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM d? s:- a- C++++ UL++++ P+ L++++ E--- W-- N o K- w-- 
O- M- V- PS+ PE Y PGP- t 5 X R- tv- b+++ DI-- D+ 
G e h-- r y** 
------END GEEK CODE BLOCK------

http://www.digriz.org.uk/jdg-qos-script/

This is in fact the link... My mistake.


I promised I will keep you informed with my progress in splitting the
bandwidth equally per IP. Unfortunately, for some odd reason Jim
diGriz''s
script did not work as I expected...

At that moment I was sure that esfq is uselsss, until I created my own
script and it worked surprisingly well :) The error as I spotted using
iptraf as a monitor was about 0.2 - 0.4 Kbytes which is very very small.

I spotted also a little "malfunction"... Maybe you can tell me
what''s wrong.

Imagine a 128 kbits (not Kbytes... a verry small) bandwidth that I have to
split for 12 clients in my LAN. Using the "default" HTB would result
in a
r2q error and a small calculated quantum. So, as Stef sugested I hardcoded
the quantum parameter for all the 12 leaf classes:

$TC class add dev $LAN_IFACE parent 1:12 classid 1:121 htb rate 8kbit ceil
96kbit quantum 1500

I noticed that changing the rate parameter will not change the guaranteed
bandwith. So here are some questions: 

1. Is "quantum" the real parameter related with the guaranteed
bandwidth?
2. Is the sum of all "leaf class quantums" relevant? (As far as I
understood
the sum of all rates in the leaf classes must not exceed the parent class
rate...)

I went further and changed the quantum from diffent clients from 1500 to
3000 (based on my knowledge this would mean double guaranteed bandwidth)

I set up a liitle test. I started 2 client computers and made them download
the same file with the same download manager. Client_1 had a 3000 quantum
and Client_2 had a 1500 quantum. The rates observed with iptraf were
10Kbytes for Client_1 and 5Kbytes for Client_2, which was the expected
result.

Another test with the quantum of Client_1 set to 15000=1500*10.
The expected result was to have a bandwith for Client_1 10 times greater
than the one for Client_2... The real results told me that whatever chenges
are made for the quantum, the Client_2 bandwith won''t fall below
4Kbytes...

3. Can you tell me why the Cllient_2 bandwidth will not decrease below 4
Kbytes?



Thanks again for your time :)

Mihai Vlad
 





  
  

-----Original Message-----
From: Fabian Gervan [mailto:fabian1@diarioc.com.ar] 
Sent: Wednesday, December 17, 2003 4:59 PM
To: Mihai Vlad
Subject: Re[2]: [LARTC] Split bandwidth equally per IP

Hello Mihai,

MV> It seems that Jim diGriz already accomplished that, but I did not read
the
MV> entire page to see that there was a script to download. My mistake :)
MV> For those who don''t know yet, here is the link:
MV> http://www.digriz.org.uk/jdg-qos-script/index.htm

404 error.
Fabian



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/