Hello, LARTC!
I have a question concerning TEQL. I have set up two Linux routers for
experimental purposes (kernel 2.4.0-test9). Every router controls 8 modems, one
of the machines is configured as a PPP dial-in server (ppp 2.4b4).
The idea is to increase throughput between both machines via TEQL queuing. This
works quite well, but unfortunately TEQL distributes the network load not as
equal as the name TEQL might suggest. If I use ICMP floods all 8 links are
uniformly under load. The same is true for UDP floods. When using single TCP
transfers (for example SCP from the SSH package) not all 8 links are used. The
packets use 3 or 4 links of the 8.
The questions is why. I have done some experiments with various applications
(FTP, NFS, SMTP, Real Audio/Video, Netmeeting, ELSA Vision). So far TEQL uses
only all 8 links when I flood the routers with packets.
Any ideas? I would like to distribute the load over all 8 links.
Here are the configs (brief, note that some options are heavily influenced by
my experiments and not meant to be used in "production" environments):
PPP options dial-in router:
----------------------------
# PPPD options
SERVER_PPP="192.168.20.1 192.168.20.2 192.168.20.3 192.168.20.4
192.168.20.5 192.168.20.6 192.168.20.7 192.168.20.8"
SERVER_NET="192.168.20.0/24"
CLIENT_PPP="192.168.00"
CLIENT_NET="192.168.0.0/24"
CLIENT="192.168.10.230"
MTU="1500"
SERVER_OPT="lock asyncmap 0 maxfail 0 idle 300 proxyarp
logfile /var/log/ppp noauth debug kdebug 3 noipdefault
persist crtscts"
MULTILINK="multilink"
IPCP_OPTS="lcp-max-configure 30 lcp-max-failure 30 ipcp-max-terminate 10
ipcp-max-configure 20 ipcp-max-failure 20"
COMP_OPTS="novj novjccomp nobsdcomp nodeflate"
SPEED=57600
After all 8 PPP daemons are initialized I set the routing:
# Activate teql0 and set routing
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
$TCC qdisc add dev lo root teql0
$IPC link set teql0 mtu $MTU up
$TCC qdisc del dev lo root teql0
$IPC route add $CLIENT_NET dev teql0
$IPC route add $CLIENT dev teql0
$IPC and $TCC is the path to the ip and the tc binary. The pppX devices are
automatically added to the TEQL queue via the PPP daemon scripts
/etc/ppp/if-up.local and /etc/ppp/if-down.local.
/etc/ppp/if-up.local (basically):
# Put link to teql0 heap
[ ${DEBUG} = "yes" ] && logger "Adding $1 to teql0
queue"
$TC qdisc add dev $1 root teql0
[ ${DEBUG} = "yes" ] && logger "Adding local address $4
to teql0"
$IP addr add $4 dev teql0
[ ${DEBUG} = "yes" ] && logger "Deleting route to remote
address $5"
$IP route del $5
/etc/ppp/if-down.local:
# Remove link from teql0 heap
[ ${DEBUG} = "yes" ] && logger "Removing $1 from teql0
queue"
$TC qdisc del dev $1 root teql0
PPP client machine:
--------------------
# PPPD options
SERVER_PPP="192.168.0.1 192.168.0.2 192.168.0.3 192.168.0.4
192.168.0.5 192.168.0.6 192.168.0.7 192.168.0.8"
SERVER_NET="192.168.0.0/24"
SERVER="192.168.10.231"
CLIENT_PPP="192.168.20"
CLIENT_NET="192.168.20.0/24"
MTU="1500"
#CLIENT_OPT="-detach lock crtscts persist demand idle 1200
noipdefault init /etc/ppp/chat/init.sh debug kdebug 2
holdoff 5 noauth novj nobsdcomp nodeflate multilink
logfile /var/log/ppp"
CLIENT_OPT="nolock asyncmap 0 maxfail 0 holdoff 2 demand persist
crtscts passive noipdefault proxyarp logfile /var/log/ppp
noauth debug kdebug 3"
PCP_OPT="lcp-max-configure 20 lcp-max-failure 20"
COMPT_OPT=""
SPEED=19200
# Get all active PPP links
cd /proc/sys/net/ipv4/conf/
PPPDEVS=`ls -1d ppp*`
# Disable RP filter
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
# Equalize links
for LINK in $PPPDEVS
do
echo "Adding $LINK..."
$TCC qdisc add dev $LINK root teql0
$IPC addr add `/usr/local/bin/getpppinfo -l $LINK` dev teql0
$IPC route del `/usr/local/bin/getpppinfo -r $LINK`
done
# Activate teql0 and set routing
$IPC link set teql0 mtu $MTU up
$IPC route add $SERVER_NET dev teql0
$IPC route add $SERVER dev teql0
Sorry for the script mess. I was glad to see it work and did not have much time
for code clean-ups.
Best regards,
René
GNU/Linux Manages! - Support, Administration, Consulting
RP3191-RIPE - Networking, Programming, Installation
On Sun, Dec 17, 2000 at 02:00:45AM +0100, Rene ''Lynx'' Pfeiffer wrote:> uniformly under load. The same is true for UDP floods. When using single TCP > transfers (for example SCP from the SSH package) not all 8 links are used. The > packets use 3 or 4 links of the 8.I think this is how it is supposed to work, I''m afraid. What happens is this. The outgoing side initially queues a few packets, but not too much. That is how TCP works - by receiving ACKnowledgement packets from the remote, the sending side knows how much data it can send. It will now wait until the remote ACKs the first three packets, say, and it will then send a couple more. The problem is that while your bandwidth has gone up, your latency hasn''t. This means that in order to use all 8 of your links, the sending side should ship 8 packets ''into the blind'', without acknowledgement. You might make this possible by raising the window size on the receiving side. However, then there is the dark problem of packet reordering. Lets say 8 packets are sent simultaneously, they can arrive in any order and it is highly unlikely that this order is 1 2 3 4 5 6 7 8. Lets say the order is 1 3 8 2 6 5 4 7, the receiving side might interpret it as follows: 1: ok -> ack up to 1 3: missed packet 2 -> ack up to 1 8: missed packet 2, 4, 5, 6, 7 -> ack up to 1 2: missed packet 4, 5, 6, 7 -> ack up to 3 6: missed packet 4, 5, 7 -> ack 3 5: missed packet 4, 7 -> ack up to 3 4: missed packet 7 -> ack up to 6 7: ack up to 7 The sending side receives the following: ack 1, ack 1, ack 1, ack 3, ack 3, ack 3, ack 6, ack 7 This will probably be interpreted as ''packet 1 got lost, lets resend it'', ''packet 3 got lost, lets resend it''. This is not the case however, but all this resending makes the sending kernel decide to slow down, and lower the ''congestion window''. This means that the kernel will not allow a lot of packets to be ''in flight'' simultaneously.> Any ideas? I would like to distribute the load over all 8 links.If you have a lot of separate streams going, it gets better. Also, if the receiving side is Linux, it might get less confused by this packet reordering. I think regular Linux is able to reorder up to 3 out of place packets.> GNU/Linux Manages! - Support, Administration, Consulting > RP3191-RIPE - Networking, Programming, InstallationGood luck! Regards, bert hubert -- PowerDNS Versatile DNS Services Trilab The Technology People ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet