LARTC - Apr 2001 - Life sucks with a saturated link.

Ok, this is where I do some thinking out loud.  After I think, I''ll
continue my research.  If and when I find the answer, I''ll write it up
nice and tidy with a mind toward submitting it to the HOWTO.  If anyone
out there wants to contribute knowledge or wisdom, then things will go
that much faster. :)

The situation is vanilla home-LAN-to-Internet, where I have no influence
over my ISP''s method of traffic control on the interface I connect to.
The majority of the traffic on the link is inbound.

The problem is that when the link is saturated with incoming traffic,
the scheduling of things does not appear to be even remotely fair.  The
usual culprit is a single long-lived TCP (HTTP) connection monopolizing
all the bandwidth.  Subsequent HTTP connections are then *very* slow to
establish and proceed, sometimes taking so long that the kernel''s
connection tracker forgets they exist and discards replies as
''un-related''.  Interactive traffic also lags severly.

So the task is to regulate traffic on the link so it gets handled in a
responsive and fair manner.  Retransmission cost for packets is
relatively expensive, so we would like to accomplish this without
randomly dropping packets[1].  The task is complicated by the fact that
the link is being saturated by incoming traffic -- traffic which we
cannot control directly.  Our saviour (I hope) is in the fact that it''s
an established TCP connection saturating the link, which we should be
able to control by regulating ACKs.

One way to obtain control over incoming traffic might be to place a
virtual entity between the incoming interface and the local network.
Now our "real" host (that is, the one which runs the services which
generate and request the traffic) gets its traffic not from the
uncontrolled modem interface, but from the virtual interface to the
virtual host.  This virtual interface is under our control, so we may
discipline incoming traffic here, and we control the traffic on the
outbound modem interface as well.

[ Sanity check:  Will our traffic policies for the virtual interface ]
[ be reflected back through the modem interface in a useful way?  Or ]
[ will excess traffic just end up piling up in the virtual host''s    ]
[ outgoing queue, not affecting upstream''s behaviour at all?         ] 
[ There''s a ''FIXME: explain congestion windows'' in
the HOWTO,        ]
[ is that the section I''m looking for?  ;)                           ]

(Note: Perhaps by the time I figure out the solution, the generalization
provided by this virtual host will not be necessary, but for now it does
make things cleaner conceptually.  Never mind the fact that I''m not
sure
how to implement it yet -- I''m confident that there is a way to do it
*somewhere* in this heap of routing power and complexity.)

Now, the problem we have here is one I have not seen addressed in
the how-to:  The outbound queue and the inbound queue are intimately
related to each other.  If the outbound queue is empty, the inbound
queue is free to use as much bandwidth as it likes.  However, if the
outbound queue contains low-latency traffic, or packets to establish
new connections, the inbound queue needs to back off enough to let this
traffic out (and to let the low-latency responses back in, though we''ll
need the ISP''s cooperation if we truly want low-latency responses at
the
head of our incoming queue).

Put another way:

Outbound and inbound traffic are really classes of a common queue[2],
and the outbound class has priority.

I say this is a problem, because my current understanding is that queues
are a per-device thing.  So how do we get these queues on different
devices to play together?

That''s about as far as my thoughts run for the time being.


Other helpful things to let me know might be:

1) Am I misdiagnosing the problem?  Is this not really a job for TC?

2) Is there some flaw in my reasoning, borne from my ignorance of how
modems schedule incoming and outgoing traffic?

3) Am I completely on crack?


Thanks for any light you can shed,

 - Kevin Turner


Endnotes:

[1] ...without randomly dropping packets:
      I think this makes the current ingress policer a poor choice.
      However, there is a patch, in iptables''s patch-o-matic
      distribution, which adds a DROPPED table to netfilter.  Perhaps
      that could be used to rescue these packets?
      (This relates to section 10.5 "FIXME: instead of dropping, can we
      assign it to a real queue?")
      Working on a patch for ingress might be another option.

[2] outbound and inbound...a common queue:
      At least, I believe this is true for modem transports.  One can
      easily imagine this might not be true for all transports.

-- 
Kevin Turner <acapnotic@users.sourceforge.net> | OpenPGP encryption
welcome here
"Uhh, it''s called LART-C.  Is this a bad sign?"

Hi,
> One way to obtain control over incoming traffic might be to place a
> virtual entity between the incoming interface and the local network.
I think we are many who wants to do that..
> [ Sanity check:  Will our traffic policies for the virtual interface ]
> [ be reflected back through the modem interface in a useful way?  Or ]
> [ will excess traffic just end up piling up in the virtual host''s 
]
> [ outgoing queue, not affecting upstream''s behaviour at all?      
]
No. In theory it will be possible to do this, but I don''t know if it is
possible in Linux.

I have for almost 2 years been running a seperate Linux box which does incomming
scheduleing for a dorm with 500 people in this way with big success. That is, a
qdisc is setup on the LAN interface of the machine and all traffic from the
internet is passing through this interface. Now, the bandwidth for this
interface has been setup to restricting internet-to-the-lan traffic to a rate a
bit lower that the capacity of the WAN link. It is described in the
documentation for the WRR qdsic. I think it will be very relevant to put some of
this stuff in the howto.
> Now, the problem we have here is one I have not seen addressed in
> the how-to:  The outbound queue and the inbound queue are intimately
> related to each other.  If the outbound queue is empty, the inbound
> queue is free to use as much bandwidth as it likes.
Is the an inbound queue? 
> However, if the
> outbound queue contains low-latency traffic, or packets to establish
> new connections, the inbound queue needs to back off enough to let this
> traffic out
I don''t understand this. To me inbound and outbound traffic is
completly unrelated. We just want to shape in both directions.
> [1] ...without randomly dropping packets:
>       I think this makes the current ingress policer a poor choice.
Yes, you won''t get any benefit from randomly dropping packets - in
effect that will just make your links lower.
> [2] outbound and inbound...a common queue:
>       At least, I believe this is true for modem transports.  One can
>       easily imagine this might not be true for all transports.
Hmm.. You can''t control when you receive a packet. At least not on
ethernet. I don''t imagine you can do that on a modem either - it is up
to the sender to send.


Christian