LARTC - Nov 2006 - Strategy for penalising IPs with too many simultaneous sessions

Hi all,

I have been trying to investigate traffic shaping in an effort to solve 
the "unfriendly network apps" problem on a test network.

I have a basis by which I''d like to shape traffic, but studying the 
howto doesn''t uncover and existing qdisc that seems to fit what I would
like to do.

The problem I would like to address is to prevent an IP address opening 
10 simultaneous streams from drowning out another IP address that opened 
1 stream.

I would like to penalise IP addresses where two or more simultaneous 
sessions are in effect, by adding a delay to the streams such that the 
total bandwidth used by the IP address is capped at a declining curve.

In other words, assuming that the data you are sending is constrained 
behind you by a 1mbps bottleneck.

When an IP has one session detected, their traffic is passed through, 
and normal rules apply.

When an IP has two sessions detected, their combined sent traffic 
towards the IP is delayed and shaped down to say 800kbps.

When an IP has three sessions detected, their combined sent traffic 
towards the IP is delayed and shaped down to say 600kbps.

The starting point of how many sessions can be open before penalising 
takes effect, the starting point of the curve and the gradient of the 
curve would obviously be subject to lots of experimentation and would be 
set by the admin.

The nett effect I am looking for, is that a user who chooses to open 
multiple simultaneous streams, should see a noticable decrease in 
maximum throughput, in an effort to discourage them from swamping the 
network with sessions.

My question is, does a qdisc exist that implements something like this?

Is this a reasonable thing to do, or will a strategy like this not work, 
and if not, why not? (for the purposes of me better understanding the 
issues).

Regards,
Graham
--

On Sat, 04 Nov 2006 02:09:03 +0200
Graham Leggett <minfrin@sharp.fm> wrote:
> Hi all,
> 
> I have been trying to investigate traffic shaping in an effort to solve 
> the "unfriendly network apps" problem on a test network.
> 
> I have a basis by which I''d like to shape traffic, but studying
the
> howto doesn''t uncover and existing qdisc that seems to fit what I
would
> like to do.
> 
> The problem I would like to address is to prevent an IP address opening 
> 10 simultaneous streams from drowning out another IP address that opened 
> 1 stream.
> 
> I would like to penalise IP addresses where two or more simultaneous 
> sessions are in effect, by adding a delay to the streams such that the 
> total bandwidth used by the IP address is capped at a declining curve.
> 
> In other words, assuming that the data you are sending is constrained 
> behind you by a 1mbps bottleneck.
> 
> When an IP has one session detected, their traffic is passed through, 
> and normal rules apply.
> 
> When an IP has two sessions detected, their combined sent traffic 
> towards the IP is delayed and shaped down to say 800kbps.
> 
> When an IP has three sessions detected, their combined sent traffic 
> towards the IP is delayed and shaped down to say 600kbps.
> 
> The starting point of how many sessions can be open before penalising 
> takes effect, the starting point of the curve and the gradient of the 
> curve would obviously be subject to lots of experimentation and would be 
> set by the admin.
> 
> The nett effect I am looking for, is that a user who chooses to open 
> multiple simultaneous streams, should see a noticable decrease in 
> maximum throughput, in an effort to discourage them from swamping the 
> network with sessions.
> 
> My question is, does a qdisc exist that implements something like this?
> 
> Is this a reasonable thing to do, or will a strategy like this not work, 
> and if not, why not? (for the purposes of me better understanding the 
> issues).
> 
> Regards,
How about making a modified version of RED that works by doing:

enqueue(skb) {
      if (qlen < min) {
	skb_queue(q, skb);
	return;
      }

      if (qlen > threshold) {
	drop(skb);
	return;
      }

       skb1 = queue[random() % qlen];
       if (skb->protocol == IP && skb1->protocol == IP &&
              skb->src.ip == skb1->src.ip) {
		drop(skb);
	        drop(skb1);
	        return;
	}
	
	skb_enqueue(q, skb);
}

There will be issues since the queue is no longer work conserving.
But it will penalize overloaders.

Graham Leggett wrote:
> Hi all,
> 
> I have been trying to investigate traffic shaping in an effort to solve 
> the "unfriendly network apps" problem on a test network.
> 
> I have a basis by which I''d like to shape traffic, but studying
the
> howto doesn''t uncover and existing qdisc that seems to fit what I
would
> like to do.
> 
> The problem I would like to address is to prevent an IP address opening 
> 10 simultaneous streams from drowning out another IP address that opened 
> 1 stream.
> 
> I would like to penalise IP addresses where two or more simultaneous 
> sessions are in effect, by adding a delay to the streams such that the 
> total bandwidth used by the IP address is capped at a declining curve.
> 
> In other words, assuming that the data you are sending is constrained 
> behind you by a 1mbps bottleneck.
> 
> When an IP has one session detected, their traffic is passed through, 
> and normal rules apply.
> 
> When an IP has two sessions detected, their combined sent traffic 
> towards the IP is delayed and shaped down to say 800kbps.
> 
> When an IP has three sessions detected, their combined sent traffic 
> towards the IP is delayed and shaped down to say 600kbps.
> 
> The starting point of how many sessions can be open before penalising 
> takes effect, the starting point of the curve and the gradient of the 
> curve would obviously be subject to lots of experimentation and would be 
> set by the admin.
> 
> The nett effect I am looking for, is that a user who chooses to open 
> multiple simultaneous streams, should see a noticable decrease in 
> maximum throughput, in an effort to discourage them from swamping the 
> network with sessions.
> 
> My question is, does a qdisc exist that implements something like this?
> 
> Is this a reasonable thing to do, or will a strategy like this not work, 
> and if not, why not? (for the purposes of me better understanding the 
> issues).
> 
> Regards,
> Graham
I''ve my misgivings with this scheme.

What you are doing makes sense only if the number of connections is a 
constrained resource. If bandwidth is the constraint, then shaping by 
source IP irrespective of number of connections will do the job. As far 
as I''ve seen, routers can support 200k connections and this is 
sufficient for many large LANs - say 500 node LAN with 400 connections 
per node.

In many cases, the user may not know how many connections he is opening 
or which app is consuming connections. Thus, the user may not be in a 
position to take remedial action and hence will be at a disadvantage.

I''m questioning the need for such a scheme really.

Mohan

Mohan Sundaram wrote:
> I''ve my misgivings with this scheme.
> 
> What you are doing makes sense only if the number of connections is a 
> constrained resource. If bandwidth is the constraint, then shaping by 
> source IP irrespective of number of connections will do the job. As far 
> as I''ve seen, routers can support 200k connections and this is 
> sufficient for many large LANs - say 500 node LAN with 400 connections 
> per node.
> 
> In many cases, the user may not know how many connections he is opening 
> or which app is consuming connections. Thus, the user may not be in a 
> position to take remedial action and hence will be at a disadvantage.
In the network in question, bandwidth is minimal (many many users 
sharing 512kbps). As a result, unlike in typical networks where 
simultaneous connections are statistically insignificant, in this case 
one user running many bittorrents can pretty much wipe out network 
performance to a ratio of 20 to 1 or more.

The typical response I have seen to this scenario is to try and 
prioritise certain protocols over others, but this strategy has the 
disadvantage of dictating to the user that they can only use those 
certain protocols.

What I would like to do instead is allow the user to use any protocol 
they like, with the caveat that attempting to open many connections 
simultaneously will result in a steadily decreasing share of the pipe, 
rather than a steadily increasing one.

Regards,
Graham
--

Graham Leggett wrote:
> In the network in question, bandwidth is minimal (many many users 
> sharing 512kbps).
You could try WRR. It doesn''t work by connection count, but works very 
well in scenario you described.

Yours sincerely,
Peter

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls

> What I would like to do instead is allow the user to use any protocol 
> they like, with the caveat that attempting to open many connections 
> simultaneously will result in a steadily decreasing share of the pipe, 
> rather than a steadily increasing one.
I solved this in a similar but slightly different way. I use connlimit
to monitor for when a user has 5 or more connections on ports above
1024. When they have, they are dropped into an ipset; all their traffic
is then monitored and any traffic on ports above 1024 is dropped to a
very low priority.

This has the advantage that web browsing they do is unaffected. Also,
it''s slightly safer than your proposed method - I have seen instances
when just normal surfing of the web can create 5 connections or more.

Something like this (eth0 is the user''s network):

iptables -t mangle -A PREROUTING -p tcp -i eth0 --dport 1024: -m \
connlimit --connlimit-above 5 -j SET --add-set p2p src

iptables -t mangle -A FORWARD -o eth0 -p tcp -m multiport --sport \
1024:65535 -m set --set p2p dst -j MARK --set-mark 60

iptables -t mangle -A FORWARD -i eth0 -p tcp -m multiport --dport \
1024:65535 -m set --set p2p src -j MARK --set-mark 60

You''ll have to compile your kernel with ipset and connlimit support.

Andy Beverley

Graham Leggett wrote:
> Mohan Sundaram wrote:
> 
>> I''ve my misgivings with this scheme.
>>
>> What you are doing makes sense only if the number of connections is a 
>> constrained resource. If bandwidth is the constraint, then shaping by 
>> source IP irrespective of number of connections will do the job. As 
>> far as I''ve seen, routers can support 200k connections and
this is
>> sufficient for many large LANs - say 500 node LAN with 400 connections 
>> per node.
>>
>> In many cases, the user may not know how many connections he is 
>> opening or which app is consuming connections. Thus, the user may not 
>> be in a position to take remedial action and hence will be at a 
>> disadvantage.
> 
> 
> In the network in question, bandwidth is minimal (many many users 
> sharing 512kbps). As a result, unlike in typical networks where 
> simultaneous connections are statistically insignificant, in this case 
> one user running many bittorrents can pretty much wipe out network 
> performance to a ratio of 20 to 1 or more.
> 
> The typical response I have seen to this scenario is to try and 
> prioritise certain protocols over others, but this strategy has the 
> disadvantage of dictating to the user that they can only use those 
> certain protocols.
> 
> What I would like to do instead is allow the user to use any protocol 
> they like, with the caveat that attempting to open many connections 
> simultaneously will result in a steadily decreasing share of the pipe, 
> rather than a steadily increasing one.
Your aims are admirable - I left my last ISP because they implemented 
app discrimination. Maybe they couldn''t set their ellacoyas to do it 
fairly - whatever.

Your situation is different - many many on 512kbs (512kbit or 4mbit?) 
either way the problem is that you are trying to shape ingress from the 
wrong end of the bottleneck which just isn''t easy or totally doable. 
Egress shouldn''t be affected by number of connections.

As you have found the more connections the harder ingress shaping 
becomes - and you may need to sacrifice 50% of your bandwidth to keep 
control.

There are things that can be tweaked regarding your setup - queue 
lengths being the main one. If you give users their own queues then you 
have to compromise - either the total is too long for your link or you 
have too short a queue for each user. ESFQ is the only thing (need to 
patch) that will do user fairness and maintain a queue len setting for 
the link (would be nice if hfsc etc could do this - may be possible to 
hack it as I see there is a drop function on leaf queues IIRC).

Another hack is to make the esfq head drop (assumes you only send 
bulk/established connections to it).

Without patching/hacking the think to do is use short queues to make 
sure you drop packets often so the tcp senders don''t flood your 
ISP/teleco buffer. On 512kbit I also used to send new connections to an 
even shorter queue (new marked using iptables connbytes), this gets them 
out of tcp slowstart quicker.

Bittorrent is slightly harder as it uses tcp full duplex for bulk - so 
acks tend to get piggybacked and stuck in egress queues meaning that non 
piggybacked acks overtake them (assuming your egress is set this way) 
and ack big chunks of a window at once causing burstiness - using short 
queues for egress bt helps this.

You are always goinng to be somewhat screwed when ingress shaping as 
nothing happens to your shaping untill it''s too late - maybe someone 
will make an ingress shaper one day that can be a bit predictive and 
back off other traffic for new connections sooner rather than later.

Andy.

As Andy muted there are commercial products that will attempt to do this
for you like Allot/Elacoya/DBAM the implementation and testing is where
this scenarios fail.

It sounds good in theory until you do it then all these odd network
errors start occurring. Cause you are now in the debate as what defines
an open or closed connection.

But here is what I know. 

Look at simplest:

dstlimit
       This module allows you to limit the packet per second (pps) rate
on  a per  destination  IP  or  per destination port base.  As opposed
to the ‘limit’ match, every destination ip / destination  port  has
it’s own limit.

Performance is a problem with the above

another iptables target is connrate you can use this to limit
connections from a set of IP addresses. So what you need to do is build
up a list of IP in a timer set and limit them however. 

recent
       Allows you to dynamically create a list of IP addresses and then
match against that list in a few different ways.

Here is a hack method of doing it imagining you are using layer7 patch.
Note you could do this using a span port if you dont want to be inline.

Every 10 minutes cat /proc/net/ip_conntrack put the output into a
database. Then use SQL group by statements to count the number of
connections per IP address src and say limit it to the TOP x%.  Put this
x% into a ipset and ratelimit the connections OR stop them creating new
connections for 10 minutes use the ctstate to stop it. Or you can be
fancy and redirect them to a webpage and until they click OK they are
still in the ipset. 

Enjoy reading about /proc/sys/.../*conntrack* params.

1 Huge double sided warning don''t deploy this until you have really
tested it. The definition of what a connection is when it is defined as
open or closed is a complex subject e.g. each ping is that a connection?
Or are you only going to limit UDP/TCP in which case what about
tunnelling. Remember if someone has a virus that creates connections
this is another problem. 


On Sat, 2006-11-11 at 12:54 +0000, Andy Furniss wrote:
> Graham Leggett wrote:
> > Mohan Sundaram wrote:
> > 
> >> I''ve my misgivings with this scheme.
> >>
> >> What you are doing makes sense only if the number of connections
is a
> >> constrained resource. If bandwidth is the constraint, then shaping
by
> >> source IP irrespective of number of connections will do the job.
As
> >> far as I''ve seen, routers can support 200k connections
and this is
> >> sufficient for many large LANs - say 500 node LAN with 400
connections
> >> per node.
> >>
> >> In many cases, the user may not know how many connections he is 
> >> opening or which app is consuming connections. Thus, the user may
not
> >> be in a position to take remedial action and hence will be at a 
> >> disadvantage.
> > 
> > 
> > In the network in question, bandwidth is minimal (many many users 
> > sharing 512kbps). As a result, unlike in typical networks where 
> > simultaneous connections are statistically insignificant, in this case
> > one user running many bittorrents can pretty much wipe out network 
> > performance to a ratio of 20 to 1 or more.
> > 
> > The typical response I have seen to this scenario is to try and 
> > prioritise certain protocols over others, but this strategy has the 
> > disadvantage of dictating to the user that they can only use those 
> > certain protocols.
> > 
> > What I would like to do instead is allow the user to use any protocol 
> > they like, with the caveat that attempting to open many connections 
> > simultaneously will result in a steadily decreasing share of the pipe,
> > rather than a steadily increasing one.
> 
> Your aims are admirable - I left my last ISP because they implemented 
> app discrimination. Maybe they couldn''t set their ellacoyas to do
it
> fairly - whatever.
> 
> Your situation is different - many many on 512kbs (512kbit or 4mbit?) 
> either way the problem is that you are trying to shape ingress from the 
> wrong end of the bottleneck which just isn''t easy or totally
doable.
> Egress shouldn''t be affected by number of connections.
> 
> As you have found the more connections the harder ingress shaping 
> becomes - and you may need to sacrifice 50% of your bandwidth to keep 
> control.
> 
> There are things that can be tweaked regarding your setup - queue 
> lengths being the main one. If you give users their own queues then you 
> have to compromise - either the total is too long for your link or you 
> have too short a queue for each user. ESFQ is the only thing (need to 
> patch) that will do user fairness and maintain a queue len setting for 
> the link (would be nice if hfsc etc could do this - may be possible to 
> hack it as I see there is a drop function on leaf queues IIRC).
> 
> Another hack is to make the esfq head drop (assumes you only send 
> bulk/established connections to it).
> 
> Without patching/hacking the think to do is use short queues to make 
> sure you drop packets often so the tcp senders don''t flood your 
> ISP/teleco buffer. On 512kbit I also used to send new connections to an 
> even shorter queue (new marked using iptables connbytes), this gets them 
> out of tcp slowstart quicker.
> 
> Bittorrent is slightly harder as it uses tcp full duplex for bulk - so 
> acks tend to get piggybacked and stuck in egress queues meaning that non 
> piggybacked acks overtake them (assuming your egress is set this way) 
> and ack big chunks of a window at once causing burstiness - using short 
> queues for egress bt helps this.
> 
> You are always goinng to be somewhat screwed when ingress shaping as 
> nothing happens to your shaping untill it''s too late - maybe
someone
> will make an ingress shaper one day that can be a bit predictive and 
> back off other traffic for new connections sooner rather than later.
> 
> Andy.
> 
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nlhttp://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Hi

On Sat, Nov 04, 2006 at 07:08:04AM +0530, Mohan Sundaram
wrote:
> What you are doing makes sense only if the number of connections is a 
> constrained resource. If bandwidth is the constraint, then shaping by 
> source IP irrespective of number of connections will do the job. As far 
> as I''ve seen, routers can support 200k connections and this is 
> sufficient for many large LANs - say 500 node LAN with 400 connections 
> per node.
I have this situation where the source IPs change on a wireless mesh.
I want fair nat. I found the fairnat script and have been reading the
lartc howto, but am not sure how to proceed.
(fairnat at http://www.metamorpher.de/fairnat/)

I want everyone to burst to (almost, a specified percentage like 90%)
full, but when all IPs are on, to share fairly irrespective of number of
connections. The IPs change dynamically. If the user doesn''t know why
their P2P or download manager is slowing them down, their problem. They can 
run whatever they want. I just don''t want one user to flood out the
rest,
and there should be a little spare so it doesn''t take time to correct
for light users.

Oh, and upload limiting (also fairly) for the lot would be good, as this
rate is limited, e.g. 512 down and 128 up. 

And it is running on freifunk on a Linksys WRT 54 GL, so performance
might be an issue on the 200Mhz processor. At the moment freifunk comes
with ingress (by internal destination IP) limiting quite nicely. The
number of clients is expected to be 0 to 64. I can deal with large
numbers of clients later, if it ever happens.

cheers,
Jan
-- 
   .~.
   /V\     Jan Groenewald
  /( )\    www.aims.ac.za
  ^^-^^