LARTC - Apr 2002 - re: Per-connection routing for multiple uplinks/providers

> I have been digging through the Lartc documentation as well as Netfilter,
 > etc. and haven''t found much on per-connection routing for
multiple
 > uplinks/providers.
 > 
 > What I would like to do is cleanly move packets out to the Internet over
 > two (maybe 3) separate interfaces, utilizing all of the bandwidth, and
 > avoiding snags.

What you (and everyone else) would really like is to make your two or
three links act like one link with bandwidth equal to the sum of the
parts.  As long as those different links have different ip addresses
(which will surely be the case if they connect to different providers)
this cannot be done.  You can indeed send traffic out in a manner that
uses all of your bandwidth (assuming your providers don''t do the
ingress/egress filtering that they should - a pretty safe bet at the
moment, sad to say).  This does introduce additional problems since
packets you send are now much more likely to arrive out of order, and
for that reason alone it''s probably not a good idea.

A more fundamental problem is that the incoming packets cannot share
the different links in the same way.  When you send a packet out you
have to choose one IP address as its source.  The reply will be sent
to that address and will have to arrive on the link with that address.
Thus, for example, if you have two links with the same incoming
bandwidth and only one connection, you can''t use more than half of
your total incoming bandwidth for that connection.

 > I could use a round-robin scheduler, which would put consecutive packets
on
 > different interfaces. I think this will run into problems when the reply
 > packets come back. Maybe not ??
As long as the provider does no filtering this will work, but will
also cause packets to arrive out of order, which is bad for
performance. 

 > I read through Arthur Leeuwen''s documentation
 >     (http://lartc.org/HOWTO//cvs/2.4routing/html/x247.html )
 > on a scheme for dividing the outgoing packets on a per-route basis.
Packets
 > going to the same destination will go through the same interface. This
gets
 > around the round-robin problem, but I think this is not
''fair'' in the sense
 > that one interface might accumulate more routes than the other, and there
 > does not seem to be a mechanism (other than periodically flushing the
route
 > tables) for evening out the flows.  It is pretty simple though and I will
 > use this as a first chop solution.
Who cares about fairness in the number of routes?  The important thing
is the bandwidth used by those routes.  And you can''t balance that,
since you don''t know when you choose the route what bandwidth will be
used by that route. 

 > Another approach to the problem would be to do a round-robin on a
 > per-connection basis. Each new connection would go out of the
''next''
 > interface.
Again, the problem is that when you have to choose you don''t know what
the bandwidth of the connection will be.  You''d do a little better
to measure the bandwidth being used currently on each link and assign
the next connection to the link with the most unused bandwidth.  But
of course, this is still only a poor approximation of what you want.

> > I have been digging through the Lartc documentation as well as
Netfilter,
> > etc. and haven''t found much on per-connection routing for
multiple
> > uplinks/providers.
> >
> > What I would like to do is cleanly move packets out to the Internet
over
> > two (maybe 3) separate interfaces, utilizing all of the bandwidth, and
> > avoiding snags.
>
>What you (and everyone else) would really like is to make your two or
>three links act like one link with bandwidth equal to the sum of the
>parts.  As long as those different links have different ip addresses
>(which will surely be the case if they connect to different providers)
>this cannot be done.
But, But, - this is really just software. We are not trying to cram wine
bottles down the internet pipe (although many would really like to do
that!).

I contend that since we are trying to send information down the pipe(s) -
ephemeral information - with no mass, only governed by Shannon''s law
(and
perhaps some other laws, but not physical laws), we can creep up on the
goal(s) I originally set out.

I am not saying that it will be easy, only that it is possible.  (Look at
the QoS success of ATM for example)

I really appreciate all of the good ideas and hints which have arrived in
response to my original plea for help.

My first steps are to construct a test lashup - something that will probe
the pipes - determine available bandwidth in both directions on each pipe,
and then tweek the connections, add queues, add QoS specs, marks, etc. to
attempt to tell the cluge what I really want to do. A lab bench engineering
setup.

Specifying what I want to do is a significant part of the problem.
Measuring how well the cluge meets the specs is also a significant part of
the solution.

It won''t be done in an instant, it will require more tools, but I think
it
can be done.  More kibitzing by folks on the lartc is very welcome.

I need some time now to absorb all of the kind comments and suggestions
which have been received so far.
>
> > Another approach to the problem would be to do a round-robin on a
> > per-connection basis. Each new connection would go out of the
''next''
> > interface.
>Again, the problem is that when you have to choose you don''t know
what
>the bandwidth of the connection will be.  You''d do a little better
>to measure the bandwidth being used currently on each link and assign
>the next connection to the link with the most unused bandwidth.  But
>of course, this is still only a poor approximation of what you want.
>
Yeah, this is a good idea.  Maybe this idea will be the ''best''
possible.

If we knew exactly how much bandwidth would be available/required for a
connection (ATM again), then there wouldn''t be any need to predict the
future.

With IP connections, the bandwidth available now isn''t really a
reliable
predictor of the bandwidth available a few milliseconds into the future
(unfortunately), so there is a bit of statistical uncertainty to meeting
goals or specifications.

From the requestee point of view, I know how much bandwidth I need to
listen to the BBC newscast, or to a company conference call. I can also
request email and ftp sessions to work in the ''background'' at
a lower
bandwidth allocation (cost?), but if I am talking to someone interactively,
it would be nice if my packets were transferred at a regular rate without
jitter or delay. IP doesn''t do this, and one can argue that it cannot.
But,
the whole thing is run by software and software can change.

Thanks again for all of the helpful comments.

BobG

Bob Gustafson writes:
 > But, But, - this is really just software. We are not trying to cram wine
 > bottles down the internet pipe (although many would really like to do
 > that!).

The limitations I point out are inherent in tcp/ip.  I think I sent a
proposal to this list describing a modification to tcp that would
allow one connection to use many ip addresses (for each endpoint).
That would allow substantial improvement, since you would be able to
switch addresses in "mid stream" (in a live connection).  It would not
solve all of the problems.  In particular, you would not be able to
efficiently use both/all addresses at once because tcp has been
adapted to work well in the case where packets arrive in order.  That
could also perhaps be overcome with changes to tcp.  Note, however,
that these changes would only help you in cases where both machines
are using the modified versions.

 > >From the requestee point of view, I know how much bandwidth I need to
 > listen to the BBC newscast, or to a company conference call. I can also
 > request email and ftp sessions to work in the
''background'' at a lower
 > bandwidth allocation (cost?), but if I am talking to someone
interactively,
 > it would be nice if my packets were transferred at a regular rate without
 > jitter or delay. IP doesn''t do this, and one can argue that it
cannot. But,
 > the whole thing is run by software and software can change.

All of the things above can already be done on a single link.
What cannot be done is make two links work like one with the 
sum of the bandwidth.

Don Cohen writes:
>Bob Gustafson writes:
> > But, But, - this is really just software. We are not trying to cram
wine
> > bottles down the internet pipe (although many would really like to do
> > that!).
>
>The limitations I point out are inherent in tcp/ip.
Yes, there are limitations in tcp/ip..
>
> > >From the requestee point of view, I know how much bandwidth I need
to
> > listen to the BBC newscast, or to a company conference call. I can
also
> > request email and ftp sessions to work in the
''background'' at a lower
> > bandwidth allocation (cost?), but if I am talking to someone
interactively,
> > it would be nice if my packets were transferred at a regular rate
without
> > jitter or delay. IP doesn''t do this, and one can argue that
it cannot. But,
> > the whole thing is run by software and software can change.
>
>All of the things above can already be done on a single link.
With IP, to get decent QoS, you need a bit of excess bandwidth on the
connection. With contention for the pipe at any point in the path from
sender to receiver, the connection reverts to a ''best efforts''
un-policed
connection.

Hmm, maybe I''m wandering into the Pub wearing a weird hat. After all,
this
is the Lartc mail list.
>What cannot be done is make two links work like one with the
>sum of the bandwidth.
I can see your point. I didn''t say that it was going to be easy.

=I just ''measured'' my DSL bandwidth at 632 kbits/sec and my
Cable bandwidth
at 897 kbits/sec  (using bandwidthplace.com)
=
BobG