LARTC - Jun 2004 - how flexible is ingress traffic policing to bandwidth limit?

[I sent this earlier but I guess the list is subscriber-only?]

I just set up wondershaper, it has a simple filter on the downstream
direction to limit the bandwidth usage:

tc qdisc add dev $DEV handle ffff: ingress
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
   0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1

This is effective but is there any way to tell it to choose only certain kinds
of streams for dropping packets? Minimally I would want to tell it to never
drop any packets in a certain list of port numbers. 

Ideally I would want it to never drop any packets that fall into the 1:10
flowid that is set up earlier in wondershaper. I''m unclear if
that''s possible
though since that flowid is set up for outgoing packets, and some of the
filters don''t even make sense for incoming packets (TOS for example).

But even if I could just tell it to never choose packets from particular
ports, that would satisfy my immediate needs.


-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Tuesday 08 June 2004 14:35, Greg Stark wrote:
> [I sent this earlier but I guess the list is subscriber-only?]
>
> I just set up wondershaper, it has a simple filter on the downstream
> direction to limit the bandwidth usage:
>
> tc qdisc add dev $DEV handle ffff: ingress
> tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
>    0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
>
> This is effective but is there any way to tell it to choose only certain
> kinds of streams for dropping packets? Minimally I would want to tell it to
> never drop any packets in a certain list of port numbers.
ingress is rather weak.  You can only really police with it.  For what you 
want most people setup the IMQ[1] device and you egress filters on it to 
control what is shaped and how.

[1] http://trash.net/~kaber/imq/

<snip>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Greg,
>>This is effective but is there any way to tell it to choose only certain
>>kinds of streams for dropping packets? Minimally I would want to tell it
to
>>never drop any packets in a certain list of port numbers.
> 
> 
> ingress is rather weak.  You can only really police with it.  For what you 
> want most people setup the IMQ[1] device and you egress filters on it to 
> control what is shaped and how.
You can create different ingress policers that only match specific ports, and
give
them different priorities, but that still won''t work as well as using
IMQ,
or if your box is a gateway (and you are only shaping traffic going through it),
then you can use egress queues on the LAN interface.

regards,

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Damion de Soto - Software Engineer  email:     damion@snapgear.com
SnapGear - A CyberGuard Company ---    ph:         +61 7 3435 2809
  | Custom Embedded Solutions          fax:         +61 7 3891 3630
  | and Security Appliances            web: http://www.snapgear.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ---  Free Embedded Linux Distro at   http://www.snapgear.org  ---
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Damion de Soto <damion@snapgear.com> writes:
> You can create different ingress policers that only match specific ports,
and
> give them different priorities, but that still won''t work as well
as using IMQ,
> or if your box is a gateway (and you are only shaping traffic going through
it),
> then you can use egress queues on the LAN interface.
For some reason that hadn''t occurred to me. That should work just fine.
I
guess I should mark the packets in iptables to avoid throttling traffic from
gateway itself, or does match see the external ip?

IMQ does seem like a handy tool, but why is there a distinction at all between
egress and ingress qdiscs at all? Why not just allow people to attach HTB as
an ingress qdisc directly?


I suppose in an ideal world the best thing would be to receive the packets,
hand them to user-space, but not mark them as received, ie, not ack them. That
avoids introducing any loss from the user point of view but still slows the
flow down. But that seems fiddly and would only work for TCP I guess.

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Greg,
> For some reason that hadn''t occurred to me. That should work just
fine. I
> guess I should mark the packets in iptables to avoid throttling traffic
from
> gateway itself, or does match see the external ip?
The only (common) time you need to use iptables to mark traffic, is when
you''re using
  egress qdiscs on your outbound (WAN) interface (to shape upload speeds) and
want to
filter based on private (LAN) IP address.
> IMQ does seem like a handy tool, but why is there a distinction at all
between
> egress and ingress qdiscs at all? Why not just allow people to attach HTB
as
> an ingress qdisc directly?
because you can''t shape inbound traffic.  Shaping works by delaying the
transmission, and
you can''t delay packets that haven''t arrived yet. Ingress
policing just drops
packets, and hopes the sender will slow down.
> I suppose in an ideal world the best thing would be to receive the packets,
> hand them to user-space, but not mark them as received, ie, not ack them.
That
> avoids introducing any loss from the user point of view but still slows the
> flow down. But that seems fiddly and would only work for TCP I guess.
That''s an interesting idea, but yeah... i think it might be a bit hard
to implement,
and violate far too many RFCs...

regards,

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Damion de Soto - Software Engineer  email:     damion@snapgear.com
SnapGear - A CyberGuard Company ---    ph:         +61 7 3435 2809
  | Custom Embedded Solutions          fax:         +61 7 3891 3630
  | and Security Appliances            web: http://www.snapgear.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ---  Free Embedded Linux Distro at   http://www.snapgear.org  ---
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Damion de Soto <damion@snapgear.com> writes:
> Greg,
> > For some reason that hadn''t occurred to me. That should work
just fine. I
> > guess I should mark the packets in iptables to avoid throttling
traffic from
> > gateway itself, or does match see the external ip?
>
> The only (common) time you need to use iptables to mark traffic, is when
> you''re using egress qdiscs on your outbound (WAN) interface (to
shape upload
> speeds) and want to filter based on private (LAN) IP address.
Well I''ll be doing the opposite here, putting an egress qdisc on my LAN
interface limiting the downstream bandwidth. So I would have the equivalent
need, to distinguish traffic originating from outside vs traffic originating
from the router so I can still scp kernels over to the router without being
limited. Hm, though in practice I hardly ever scp things _from_ the router so
perhaps I don''t care.
> > IMQ does seem like a handy tool, but why is there a distinction at all
between
> > egress and ingress qdiscs at all? Why not just allow people to attach
HTB as
> > an ingress qdisc directly?
>
> because you can''t shape inbound traffic. Shaping works by delaying
the
> transmission, and you can''t delay packets that haven''t
arrived yet. Ingress
> policing just drops packets, and hopes the sender will slow down.
Well ultimately all shaping works by dropping packets. Merely delaying
transmission isn''t going to slow down anything in the long run, just
increase
the pipeline. You can delay and/or drop them after they''ve arrived just
as
easily. Though it would have to be before they''re ack''d and
delivered to the
user. That''s basically what IMQ does, I''m just saying perhaps
that should just
work instead of requiring a fake interface.

Hm, I wonder if I want RED or something similar to ensure packets get dropped
fast enough instead of filling HTB queues and then dropping.
> > I suppose in an ideal world the best thing would be to receive the
packets,
> > hand them to user-space, but not mark them as received, ie, not ack
them. That
> > avoids introducing any loss from the user point of view but still
slows the
> > flow down. But that seems fiddly and would only work for TCP I guess.
>
> That''s an interesting idea, but yeah... i think it might be a bit
hard to
> implement, and violate far too many RFCs...
My original thought was to just drop an ACK packet, which would clearly not
violate any assumptions. But if there''s data in the ACK packet that
could have
negative effects. I suspect this lies well into Bad Idea(tm) realm, I''m
just
thinking out loud.

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Tuesday 08 June 2004 23:33, Greg Stark wrote:
> Damion de Soto <damion@snapgear.com> writes:
<snip>
> >
> > because you can''t shape inbound traffic. Shaping works by
delaying the
> > transmission, and you can''t delay packets that
haven''t arrived yet.
> > Ingress policing just drops packets, and hopes the sender will slow
down.
>
> Well ultimately all shaping works by dropping packets. Merely delaying
> transmission isn''t going to slow down anything in the long run,
just
> increase the pipeline. You can delay and/or drop them after
they''ve arrived
> just as easily. Though it would have to be before they''re
ack''d and
> delivered to the user. That''s basically what IMQ does,
I''m just saying
> perhaps that should just work instead of requiring a fake interface.
Ultimately, packets from a misbehaving flow can be dropped, but it does not 
always come to a drop.  When you shape on egress, you force applications on 
the local network to throttle back, believing they''re sending as fast
as the
receiver can receive.  As you delay, TCP figures it out.  Contrast that with 
ingress, where the packets you want to delay are already on their way.
> Hm, I wonder if I want RED or something similar to ensure packets get
> dropped fast enough instead of filling HTB queues and then dropping.
If you''re curious about RED, here''s a possible example
implementation for
ingress policing:

http://digriz.org.uk/jdg-qos-script/

<snip>

-- 

Jason Boxman
Perl Programmer / *NIX Systems Administrator
Shimberg Center for Affordable Housing | University of Florida
http://edseek.com/ - Linux and FOSS stuff

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Jason Boxman <jasonb@edseek.com> writes:
> On Tuesday 08 June 2004 23:33, Greg Stark wrote:
> >
> > Well ultimately all shaping works by dropping packets. Merely delaying
> > transmission isn''t going to slow down anything in the long
run, just
> > increase the pipeline. You can delay and/or drop them after
they''ve arrived
> > just as easily. Though it would have to be before they''re
ack''d and
> > delivered to the user. That''s basically what IMQ does,
I''m just saying
> > perhaps that should just work instead of requiring a fake interface.
> 
> Ultimately, packets from a misbehaving flow can be dropped, but it does not
> always come to a drop.  When you shape on egress, you force applications on
> the local network to throttle back, believing they''re sending as
fast as the
> receiver can receive.  As you delay, TCP figures it out.  
I don''t think so. It may look like that''s what''s
happening, but at least for
TCP I don''t think it works that way. As long as packets aren''t
being dropped
then TCP will just continue to grow the window, interpreting this delay as
just a longer pipeline that needs filling. Applications will be slowed down
temporarily because it takes time to do this, but they''ll eventually be
outputting data just as fast as an application without a shaper.

Only when packets get dropped or are delayed so long that the client
retransmits does TCP scale back the transmit window. And only when that
happens does the client see any reduced bandwidth. So if your shaper
isn''t
dropping packets it''s just evening out the flow of data, not actually
affecting the net rate the clients can pump out data.
> Contrast that with ingress, where the packets you want to delay are already
> on their way.
Well on egress the packets are "already on their way" as well, after
all.
They''re just haven''t gone as many hops. Even for locally
generated traffic the
egress qdisc is being run after the data packetized and ready to go.
> > Hm, I wonder if I want RED or something similar to ensure packets get
> > dropped fast enough instead of filling HTB queues and then dropping.
> 
> If you''re curious about RED, here''s a possible example
implementation for
> ingress policing:
> 
> http://digriz.org.uk/jdg-qos-script/
This is an interesting script. It looks like a successor to wondershaper. But
I''m a bit too deep in my own re-implementation of wondershaper now to
start
over.

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Wed, 2004-06-09 at 09:03, Greg Stark wrote:
> Damion de Soto <damion@snapgear.com> writes:
> 
> > because you can''t shape inbound traffic. Shaping works by
delaying the
> > transmission, and you can''t delay packets that
haven''t arrived yet. Ingress
> > policing just drops packets, and hopes the sender will slow down.
>
Sorry to interrupt the flow, especially being a newbie, but won´t the
sender just retransmit the dropped packets at the same rate? I am not so
thorogh with TCP/IP, but is there something in the protocol that speeds
or slows the transmission.

Please do explain in the TCP/IP for complete idiots terminology ;-))

With best regards.
Sanjay.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Sanjay Arora <skpobox@gawab.com> writes:
> Sorry to interrupt the flow, especially being a newbie, but won´t the
> sender just retransmit the dropped packets at the same rate? 
no.
> I am not so thorogh with TCP/IP, but is there something in the protocol
that
> speeds or slows the transmission.
yes.
> Please do explain in the TCP/IP for complete idiots terminology ;-))
You could do worse than this:

http://www.thinkingsecure.com/docs/TCPIP-Illustrated-1/tcp_time.htm#21_0

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Wednesday 09 June 2004 16:09, Greg Stark wrote:
> Sanjay Arora <skpobox@gawab.com> writes:
> > Sorry to interrupt the flow, especially being a newbie, but won´t the
> > sender just retransmit the dropped packets at the same rate?
>
> no.
>
> > I am not so thorogh with TCP/IP, but is there something in the
protocol
> > that speeds or slows the transmission.
>
> yes.
>
> > Please do explain in the TCP/IP for complete idiots terminology ;-))
>
> You could do worse than this:
>
> http://www.thinkingsecure.com/docs/TCPIP-Illustrated-1/tcp_time.htm#21_0
I have to admit, I am thus far thoroughly enjoying:

Jacobson, V., Congestion Avoidance and Control. Proceedings of ACM SIGCOMM 
''88. August 1988, p. 314-329.

It''s also available in Postscript format:

ftp://ftp.ee.lbl.gov/papers/congavoid.ps.Z

For those who don''t know how TCP handles congestion, this is a rather 
excellent read.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Greg Stark wrote:
> Jason Boxman <jasonb@edseek.com> writes:
> 
> 
>>On Tuesday 08 June 2004 23:33, Greg Stark wrote:
>>
>>>Well ultimately all shaping works by dropping packets. Merely
delaying
>>>transmission isn''t going to slow down anything in the long
run, just
>>>increase the pipeline. You can delay and/or drop them after
they''ve arrived
>>>just as easily. Though it would have to be before they''re
ack''d and
>>>delivered to the user. That''s basically what IMQ does,
I''m just saying
>>>perhaps that should just work instead of requiring a fake interface.
>>
>>Ultimately, packets from a misbehaving flow can be dropped, but it does
not
>>always come to a drop.  When you shape on egress, you force applications
on
>>the local network to throttle back, believing they''re sending
as fast as the
>>receiver can receive.  As you delay, TCP figures it out.  
> 
> 
> I don''t think so. It may look like that''s what''s
happening, but at least for
> TCP I don''t think it works that way. As long as packets
aren''t being dropped
> then TCP will just continue to grow the window, interpreting this delay as
> just a longer pipeline that needs filling. Applications will be slowed down
> temporarily because it takes time to do this, but they''ll
eventually be
> outputting data just as fast as an application without a shaper.
Eventually in the egress case either you will run out of buffer and 
drop, or your queue is big enough for the advertised window (say 
32k-64k) and tcp will only add a packet for everyone acked - you decide 
when the packets go so you have good control. If you drop, the sender 
reduces it''s cwin and only slowly tries to increase until another drop.
> 
> Only when packets get dropped or are delayed so long that the client
> retransmits does TCP scale back the transmit window. And only when that
> happens does the client see any reduced bandwidth. So if your shaper
isn''t
> dropping packets it''s just evening out the flow of data, not
actually
> affecting the net rate the clients can pump out data.
Once you have dropped the rate you release packets does controll the 
flow and if it''s slow the cwin will only grow slowly as it''s
clocked by
acks (I think).
> 
> 
>>Contrast that with ingress, where the packets you want to delay are
already
>>on their way.
> 
> 
> Well on egress the packets are "already on their way" as well,
after all.
> They''re just haven''t gone as many hops. Even for locally
generated traffic the
> egress qdisc is being run after the data packetized and ready to go.
> 
I think what Jason was getting at is that on egress you have total 
control - it doesn''t matter what TCP does.

Ingress shaping is harder as packets are already headed for you - in the 
case of P2P possibly quite alot - it''s easier to shape when peering
with
"real" servers. Though in both cases, the fact that TCP slow (but sort
of exponential) start overshoots bandwidth hurts ingress shaping, but 
not egress.

Andy.
> 
>>>Hm, I wonder if I want RED or something similar to ensure packets
get
>>>dropped fast enough instead of filling HTB queues and then dropping.
>>
>>If you''re curious about RED, here''s a possible example
implementation for
>>ingress policing:
>>
>>http://digriz.org.uk/jdg-qos-script/
> 
> 
> This is an interesting script. It looks like a successor to wondershaper.
But
> I''m a bit too deep in my own re-implementation of wondershaper now
to start
> over.
> 

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/