LARTC - Nov 2004 - Shaping traffic on heavily oversubscribed links?

Hi all,

I am having some fun with traffic shaping, and have run into an 
interesting situation.  Here is South Africa, most internet links are 
heavily oversubscribed, which means that in most cases the local link is 
_not_ the bottleneck, and shaping on the local link does not help that 
much...

Does anybody have some tips on shaping such links?  How can you get 
interractive traffic if you don''t know how much bandwidth to reserve
for
it? How can you give fair access to a link if you don''t know what the 
link capacity is?

Are there perhaps some tools to monitor retransmissions to try and 
determine congestion levels, and from that adjust shaped bandwidth?

Am I perhaps missing something simple in this scenario?

Thanks!
-justin
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Quick answer is: you can''t.  You need to know the bandwidth so that you
can
control the queue.

You bring up some interesting theoretical ideas about monitoring the overall 
congestion level and forcing back-offs when you sense that you''re
reaching
the current level, though.  That would sort of involve dynamically adjusting 
the shaping rules.  As far as I know there is nothing in existance that does 
this, but its an interesting idea to think about.  If you were going to try 
something like this I think you''d need some sort of reliable indicator
of
what the current congestion is like... perhaps some steady ping to use as 
your "canary in the coal mine".  Then set up several shaping scripts
that
assume different levels of bandwidth, and depending on the current "canary 
ping", either upgrade or downgrade your assumed bandwidth by calling the 
appropriate script.  So I guess if you wanted to try this, maybe you could 
set up three scripts, one set to 80% of your bandwidth, one set to 50%, and 
one set to 20%.  Then set up a cron job that checks the "ping" for
your high
priority traffic, and calls the appropriate script to adjust.  I doubt
it''ll
work, but it could be an interesting experiment.

----- Original Message ----- 
From: "Justin Schoeman" <justin@expertron.co.za>
To: <lartc@mailman.ds9a.nl>
Sent: Thursday, November 25, 2004 3:17 AM
Subject: [LARTC] Shaping traffic on heavily oversubscribed links?

> Hi all,
>
> I am having some fun with traffic shaping, and have run into an 
> interesting situation.  Here is South Africa, most internet links are 
> heavily oversubscribed, which means that in most cases the local link is 
> _not_ the bottleneck, and shaping on the local link does not help that 
> much...
>
> Does anybody have some tips on shaping such links?  How can you get 
> interractive traffic if you don''t know how much bandwidth to
reserve for
> it? How can you give fair access to a link if you don''t know what
the link
> capacity is?
>
> Are there perhaps some tools to monitor retransmissions to try and 
> determine congestion levels, and from that adjust shaped bandwidth?
>
> Am I perhaps missing something simple in this scenario?
>
> Thanks!
> -justin
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

it''s worse than that. we faced the same problems in china. an 
oversubscribed adsl system. in fact there is more than one problem with 
the public internet in these scenarios. the first as you have identified 
is the lack of capacity. then you can''t control the downstream routing
-
eg traffic from china to australia was going to the west coast us, then 
to florida and finally back to australia via a different gateway in la! 
some of the routers along the way have large buffers meaning that they 
can introduce significant delays into the transmission. and finally when 
all these things overflow you get large packet loss - up to 50% at some 
times.

the net result is this: high packet loss means forget about real time 
services such as voip or video conferencing. they are asynchronous 
services that don''t retransmit and are also not very tolerant of packet
loss (over about 10%). long ping times means that synchronous services 
such as email are very slow, coupled with packet loss you might as well 
be on a dial up.

our solution in a commercial environment was to keep upgrading until we 
found a service level that works (more or less). we moved from adsl to 
fibre. found an isp at each end that we could talk to and who could give 
assistance with routing issues. eg if they have multiple gateways we 
experimented until we found the most efficient gateway.

we also know of larger comapnies who pay substantial fees to achieve end 
to end qos.

all of this means there''s not a lot you can do at your end with limited
choices except to accept that there is no bandwidth at some times of the 
day and go have a beer instead :)

regards

rick

Chris Bennett wrote:
> Quick answer is: you can''t.  You need to know the bandwidth so
that
> you can control the queue.
>
> You bring up some interesting theoretical ideas about monitoring the 
> overall congestion level and forcing back-offs when you sense that 
> you''re reaching the current level, though.  That would sort of
involve
> dynamically adjusting the shaping rules.  As far as I know there is 
> nothing in existance that does this, but its an interesting idea to 
> think about.  If you were going to try something like this I think 
> you''d need some sort of reliable indicator of what the current 
> congestion is like... perhaps some steady ping to use as your "canary 
> in the coal mine".  Then set up several shaping scripts that assume 
> different levels of bandwidth, and depending on the current "canary 
> ping", either upgrade or downgrade your assumed bandwidth by calling 
> the appropriate script.  So I guess if you wanted to try this, maybe 
> you could set up three scripts, one set to 80% of your bandwidth, one 
> set to 50%, and one set to 20%.  Then set up a cron job that checks 
> the "ping" for your high priority traffic, and calls the
appropriate
> script to adjust.  I doubt it''ll work, but it could be an
interesting
> experiment.
>
> ----- Original Message ----- From: "Justin Schoeman" 
> <justin@expertron.co.za>
> To: <lartc@mailman.ds9a.nl>
> Sent: Thursday, November 25, 2004 3:17 AM
> Subject: [LARTC] Shaping traffic on heavily oversubscribed links?
>
>
>> Hi all,
>>
>> I am having some fun with traffic shaping, and have run into an 
>> interesting situation.  Here is South Africa, most internet links are 
>> heavily oversubscribed, which means that in most cases the local link 
>> is _not_ the bottleneck, and shaping on the local link does not help 
>> that much...
>>
>> Does anybody have some tips on shaping such links?  How can you get 
>> interractive traffic if you don''t know how much bandwidth to
reserve
>> for it? How can you give fair access to a link if you don''t
know what
>> the link capacity is?
>>
>> Are there perhaps some tools to monitor retransmissions to try and 
>> determine congestion levels, and from that adjust shaped bandwidth?
>>
>> Am I perhaps missing something simple in this scenario?
>>
>> Thanks!
>> -justin
>> _______________________________________________
>> LARTC mailing list / LARTC@mailman.ds9a.nl
>> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>>
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Thanks everybody for your advice... This is going to be an interesting 
one to try and solve ;-).

-justin

Justin Schoeman wrote:
> Hi all,
> 
> I am having some fun with traffic shaping, and have run into an 
> interesting situation.  Here is South Africa, most internet links are 
> heavily oversubscribed, which means that in most cases the local link is 
> _not_ the bottleneck, and shaping on the local link does not help that 
> much...
> 
> Does anybody have some tips on shaping such links?  How can you get 
> interractive traffic if you don''t know how much bandwidth to
reserve for
> it? How can you give fair access to a link if you don''t know what
the
> link capacity is?
> 
> Are there perhaps some tools to monitor retransmissions to try and 
> determine congestion levels, and from that adjust shaped bandwidth?
> 
> Am I perhaps missing something simple in this scenario?
> 
> Thanks!
> -justin
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Thursday 25 November 2004 13:01, Chris Bennett wrote:
> Quick answer is: you can''t.  You need to know the bandwidth so
that you can
> control the queue.
Indeed.  I suffer from the same problem with my PPPoATM link, where my shaping 
configuration assumes I''m operating over an Ethernet link when
it''s really
encapsulated over an ATM network.  Bandwidth available for Ethernet packets 
varies depending on how many packets are going out over the link and the size 
of each packet.

-- 

Jason Boxman
Perl Programmer / *NIX Systems Administrator
Shimberg Center for Affordable Housing | University of Florida
http://edseek.com/ - Linux and FOSS stuff

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Justin Schoeman wrote:
> Hi all,
>
> I am having some fun with traffic shaping, and have run into an 
> interesting situation.  Here is South Africa, most internet links are 
> heavily oversubscribed, which means that in most cases the local link 
> is _not_ the bottleneck, and shaping on the local link does not help 
> that much...
>
We have the same problem in DSL lines here in Greece.

I have found that while the average efective speed on such lines varies, 
tha average rate of packets is more or less constant. I have a theory 
for this. I believe that the routers that forward the traffic on 
congested lines - on ISPs and on the ATM circuits at the telcoms -
don''t
take the extra time needed to calculate the size of the packets and 
distribute the traffic on a per packet basis. This leads to a
''fairness''
among the end receivers based on packets/sec instead of  bandwidth.

To be more specific. In my ADSL line I usually achieve between 20-30 pps 
(measured with MRTG). With an average packet size of 1500 this is 20-45 
kbytes/sec. But packets sizes close to the MTU are found on single 
ftp/http connections and pretty much nowehere else. Packet sizes of 400 
to 500 are more realistic, especially when p2p programs are involved. 
20-30 packets give 8-10kbytes/sec. You can expect even less when using 
voip programs which utilize smaller packets.

If you find that single a FTP session tends to get more bandwidth thatn 
p2p programs or multiuser traffic then you have a simillar problem to 
our own. I would suggest that you setup MRTG to monitor packets to 
research further into this.


> Does anybody have some tips on shaping such links?  How can you get 
> interractive traffic if you don''t know how much bandwidth to
reserve
> for it? How can you give fair access to a link if you don''t know
what
> the link capacity is?
>
Well, I am working on one. Since I can''t shape bandwidth because it 
flactuates erratically with time and usage I decided to shape packets. I 
have created a new queueing discipline based on TBF which uses packets 
instead of bytes for its tokens and I am allocating a constant 
packet/sec rate on each user of my ADSL line. A better solution would be 
to create an HTB alike packet-based qdisc for dynamic shaping.

If you find that you have the some problem as me and you want to 
experiment with a packet-based TBF qdisc I can send you a patch for 
linux-2.6.8 and iproute2 in this list.

I would like to here your thought on this anyway ...

Dimitris



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Dimitris Kotsonis wrote:
> Justin Schoeman wrote:
> 
>> Hi all,
>>
>> I am having some fun with traffic shaping, and have run into an 
>> interesting situation.  Here is South Africa, most internet links are 
>> heavily oversubscribed, which means that in most cases the local link 
>> is _not_ the bottleneck, and shaping on the local link does not help 
>> that much...
>>
> 
> We have the same problem in DSL lines here in Greece.
> 
> I have found that while the average efective speed on such lines varies, 
> tha average rate of packets is more or less constant. I have a theory 
> for this. I believe that the routers that forward the traffic on 
> congested lines - on ISPs and on the ATM circuits at the telcoms -
don''t
> take the extra time needed to calculate the size of the packets and 
> distribute the traffic on a per packet basis. This leads to a
''fairness''
> among the end receivers based on packets/sec instead of  bandwidth.
> 
> To be more specific. In my ADSL line I usually achieve between 20-30 pps 
> (measured with MRTG). With an average packet size of 1500 this is 20-45 
> kbytes/sec. But packets sizes close to the MTU are found on single 
> ftp/http connections and pretty much nowehere else. Packet sizes of 400 
> to 500 are more realistic, especially when p2p programs are involved. 
> 20-30 packets give 8-10kbytes/sec. You can expect even less when using 
> voip programs which utilize smaller packets.
> 
> If you find that single a FTP session tends to get more bandwidth thatn 
> p2p programs or multiuser traffic then you have a simillar problem to 
> our own. I would suggest that you setup MRTG to monitor packets to 
> research further into this.
It is normal for an FTP download to take over from p2ps the latter are 
likely to be higher latency, so TCP will let a lower latency FTP grab 
more bandwidth.

Try shaping with HTB and sfq - It should help.
> 
> 
> 
>> Does anybody have some tips on shaping such links?  How can you get 
>> interractive traffic if you don''t know how much bandwidth to
reserve
>> for it? How can you give fair access to a link if you don''t
know what
>> the link capacity is?
>>
> 
> Well, I am working on one. Since I can''t shape bandwidth because
it
> flactuates erratically with time and usage I decided to shape packets. I 
> have created a new queueing discipline based on TBF which uses packets 
> instead of bytes for its tokens and I am allocating a constant 
> packet/sec rate on each user of my ADSL line. A better solution would be 
> to create an HTB alike packet-based qdisc for dynamic shaping.
> 
> If you find that you have the some problem as me and you want to 
> experiment with a packet-based TBF qdisc I can send you a patch for 
> linux-2.6.8 and iproute2 in this list.
> 
> I would like to here your thought on this anyway ...
> 
> Dimitris
> 
> 
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/