Miguel Ángel Domínguez Durán
2005-Feb-11 12:56 UTC
Help!!! Bandwith Control with a NAT machine
Hello everyone,
First of all, sorry for my poor english.
I''ve been working with this for a few weeks and I''m getting
sick...
I''m trying to control the bandwith in my network using the following
script. The machine where the script is running makes NAT, eth0 is connected to
the router and eth1 is connected to the Lan. When I run the script it
doesn''t appear any errors, i have recompiled a Red Hat kernel 2.4.20,
check all the options right and installed iproute2-2.6.9. The result is that
every packet is sent to the default queue and I can''t understand why.
It seems like iptables is not marking any of the packets, all the queues and
classes are empty, traffic always goes through default queues in uplink and
downlink.
Here is the script, which is a modification of some things i''ve found
in the net:
#!/bin/bash
#
#
DEV1=eth1 #salida a red local
DEV0=eth0 #salida a internet
#
TC=/usr/sbin/tc
if [ "$1" = "status" ]
then
echo "Enlace descendente"
echo "[qdisc]"
$TC -s qdisc show dev $DEV1
echo "[class]"
$TC -s class show dev $DEV1
echo "[filter]"
$TC -s filter show dev $DEV1
echo "Enlace ascendente"
echo "[qdisc]"
$TC -s qdisc show dev $DEV0
echo "[class]"
$TC -s class show dev $DEV0
echo "[filter]"
$TC -s filter show dev $DEV0
# echo "[iptables]"
# iptables -t mangle -L MYSHAPER-OUT -v -x 2> /dev/null
# iptables -t mangle -L MYSHAPER-IN -v -x 2> /dev/null
exit
fi
# Reset everything to a known state (cleared)
$TC qdisc del dev $DEV0 root 2> /dev/null > /dev/null
$TC qdisc del dev $DEV1 root 2> /dev/null > /dev/null
iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-OUT 2> /dev/null >
/dev/null
iptables -t mangle -F MYSHAPER-OUT 2> /dev/null > /dev/null
iptables -t mangle -X MYSHAPER-OUT 2> /dev/null > /dev/null
iptables -t mangle -D PREROUTING -i $DEV1 -j MYSHAPER-IN 2> /dev/null >
/dev/null
iptables -t mangle -F MYSHAPER-IN 2> /dev/null > /dev/null
iptables -t mangle -X MYSHAPER-IN 2> /dev/null > /dev/null
#iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-IN 2> /dev/null >
/dev/null
if [ "$1" = "stop" ]
then
echo "Shaping removed on $DEV1."
echo "Shaping removed on $DEV0."
exit
fi
###########################################################
#
# Inbound Shaping (limits total bandwidth to 1000Kbps)
# Este es el enlace descendente, desde internet hacia la red interna de
Cherrytel
# set queue size to give latency of about 2 seconds on low-prio packets
ip link set dev $DEV1 qlen 30
# changes mtu on the outbound device. Lowering the mtu will result
# in lower latency but will also cause slightly lower throughput due
# to IP and TCP protocol overhead.
ip link set dev $DEV1 mtu 1000
# add HTB root qdisc
$TC qdisc add dev $DEV1 root handle 1: htb default 37
# add main rate limit classes
$TC class add dev $DEV1 parent 1: classid 1:1 htb rate 1000kbit
# add leaf classes - We grant each class at LEAST it''s "fair
share" of bandwidth.
# this way no class will ever be starved by another class.
Each
# class is also permitted to consume all of the available
bandwidth
# if no other classes are in use.
$TC class add dev $DEV1 parent 1:1 classid 1:20 htb rate 64kbit ceil 1000kbit
$TC class add dev $DEV1 parent 1:1 classid 1:21 htb rate 64kbit ceil 1000kbit
$TC class add dev $DEV1 parent 1:1 classid 1:22 htb rate 64kbit ceil 1000kbit
$TC class add dev $DEV1 parent 1:1 classid 1:37 htb rate 832kbit ceil 1000kbit
#por defecto
$TC class add dev $DEV1 parent 1:1 classid 1:23 htb rate 64kbit ceil 64kbit
#prueba, maq WiFi
# attach qdisc to leaf classes - here we at SFQ to each priority class. SFQ
insures that
# within each class connections will be treated
(almost) fairly.
$TC qdisc add dev $DEV1 parent 1:20 handle 20: sfq perturb 10
$TC qdisc add dev $DEV1 parent 1:21 handle 21: sfq perturb 10
$TC qdisc add dev $DEV1 parent 1:22 handle 22: sfq perturb 10
$TC qdisc add dev $DEV1 parent 1:37 handle 37: sfq perturb 10
$TC qdisc add dev $DEV1 parent 1:23 handle 23: sfq perturb 10
# filter traffic into classes by fwmark - here we direct traffic into priority
class according to
# the fwmark set on the packet (we set
fwmark with iptables
# later). Note that above
we''ve set the default priority
# class to 1:37 so unmarked packets (or
packets marked with
# unfamiliar IDs) will be defaulted to
the lowest priority
# class.
$TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 20 fw flowid 1:20
$TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 21 fw flowid 1:21
$TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 22 fw flowid 1:22
$TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 23 fw flowid 1:23
# Marking the packets. Se marcan los paquetes en el interfaz contrario, para que
no se vean
# afectados por el NAT que hacen las reglas del firewall
iptables -t mangle -N MYSHAPER-OUT
iptables -t mangle -I PREROUTING -i $DEV0 -j MYSHAPER-OUT
#iptables -t mangle -A MYSHAPER-IN -p tcp --sport ssh -j MARK --set-mark 20
iptables -A MYSHAPER-OUT -d 172.9.264.30 -t mangle -j MARK --set-mark 20
iptables -A MYSHAPER-OUT -d 172.9.264.31 -t mangle -j MARK --set-mark 20
iptables -A MYSHAPER-OUT -d 172.9.264.32 -t mangle -j MARK --set-mark 20
iptables -A MYSHAPER-OUT -d 172.9.234.22 -t mangle -j MARK --set-mark 21
iptables -A MYSHAPER-OUT -d 172.9.234.71 -t mangle -j MARK --set-mark 21
iptables -A MYSHAPER-OUT -d 172.9.234.25 -t mangle -j MARK --set-mark 22
iptables -A MYSHAPER-OUT -d 172.9.234.14 -t mangle -j MARK --set-mark 23
# redundant- mark any unmarked packets as 26 (low prio)
#El resto de tráco irÃal flujo por defecto, el 2:37.
# Done with inbound shaping
#
####################################################
echo "Control del enlace descendente activado."
#Si solo se desea controlar el enlace descendente, quitar el comentario de la
siguiente instruccion exit
#exit
###########################################################
#
# Outbound Shaping (limits total bandwidth to 1000Kbps)
# Este es el enlace ascendente, desde la red interna de Cherrytel a internet
# set queue size to give latency of about 2 seconds on low-prio packets
ip link set dev $DEV0 qlen 30
# changes mtu on the outbound device. Lowering the mtu will result
# in lower latency but will also cause slightly lower throughput due
# to IP and TCP protocol overhead.
ip link set dev $DEV0 mtu 1000
# add HTB root qdisc
$TC qdisc add dev $DEV0 root handle 2: htb default 73
# add main rate limit classes
$TC class add dev $DEV0 parent 2: classid 2:1 htb rate 1000kbit
# add leaf classes - We grant each class at LEAST it''s "fair
share" of bandwidth.
# this way no class will ever be starved by another class.
Each
# class is also permitted to consume all of the available
bandwidth
# if no other classes are in use.
$TC class add dev $DEV0 parent 2:1 classid 2:70 htb rate 64kbit ceil 1000kbit
$TC class add dev $DEV0 parent 2:1 classid 2:71 htb rate 64kbit ceil 1000kbit
$TC class add dev $DEV0 parent 2:1 classid 2:72 htb rate 64kbit ceil 1000kbit
$TC class add dev $DEV0 parent 2:1 classid 2:87 htb rate 744kbit ceil 1000kbit
$TC class add dev $DEV0 parent 2:1 classid 2:73 htb rate 64kbit ceil 64kbit
#prueba
# attach qdisc to leaf classes - here we at SFQ to each priority class. SFQ
insures that
# within each class connections will be treated
(almost) fairly.
$TC qdisc add dev $DEV0 parent 2:70 handle 70: sfq perturb 10
$TC qdisc add dev $DEV0 parent 2:71 handle 71: sfq perturb 10
$TC qdisc add dev $DEV0 parent 2:72 handle 72: sfq perturb 10
$TC qdisc add dev $DEV0 parent 2:87 handle 87: sfq perturb 10
$TC qdisc add dev $DEV0 parent 2:73 handle 73: sfq perturb 10
# filter traffic into classes by fwmark - here we direct traffic into priority
class according to
# the fwmark set on the packet (we set
fwmark with iptables
# later). Note that above
we''ve set the default priority
# class to 1:87 so unmarked packets (or
packets marked with
# unfamiliar IDs) will be defaulted to
the lowest priority
# class.
$TC filter add dev $DEV0 parent 2:0 prio 1 protocol ip handle 70 fw flowid 1:70
$TC filter add dev $DEV0 parent 2:0 prio 2 protocol ip handle 71 fw flowid 1:71
$TC filter add dev $DEV0 parent 2:0 prio 3 protocol ip handle 72 fw flowid 1:72
$TC filter add dev $DEV0 parent 2:0 prio 4 protocol ip handle 73 fw flowid 1:73
# Marking the packets. Se marcan los paquetes en el interfaz contrario, para que
no se vean
# afectados por el NAT que hacen las reglas del firewall
iptables -t mangle -N MYSHAPER-IN
iptables -t mangle -I PREROUTING -i $DEV1 -j MYSHAPER-IN
#iptables -t mangle -A MYSHAPER-IN -p ! tcp -j MARK --set-mark 20
iptables -A MYSHAPER-IN -s 172.9.234.30 -t mangle -j MARK --set-mark 70
iptables -A MYSHAPER-IN -s 172.9.234.31 -t mangle -j MARK --set-mark 70
iptables -A MYSHAPER-IN -s 172.9.234.32 -t mangle -j MARK --set-mark 70
iptables -A MYSHAPER-IN -s 172.9.234.22 -t mangle -j MARK --set-mark 71
iptables -A MYSHAPER-IN -s 172.9.234.71 -t mangle -j MARK --set-mark 71
iptables -A MYSHAPER-IN -s 172.9.234.25 -t mangle -j MARK --set-mark 72
#Prueba maquina WiFi
iptables -A MYSHAPER-IN -s 172.9.234.14 -t mangle -j MARK --set-mark 73
#El resto de tráco irÃal flujo por defecto, el 2:87.
# Done with outbound shaping
####################################################
echo "Control del enlace ascendente activado."
exit
Thanks for your help!
UN CORDIAL SALUDO
Miguel Ángel Domínguez Durán.
Departamento Técnico.
Cherrytel Comunicaciones, S.L.
mdominguez@cherrytel.com
http://www.cherrytel.com/
Tlf. 902 115 673
Fax 952218170
Miguel Ángel Domínguez Durán wrote:> Hello everyone, > First of all, sorry for my poor english. > I''ve been working with this for a few weeks and I''m getting sick... > I''m trying to control the bandwith in my network using the following script. The machine where the script is running makes NAT, eth0 is connected to the router and eth1 is connected to the Lan. When I run the script it doesn''t appear any errors, i have recompiled a Red Hat kernel 2.4.20, check all the options right and installed iproute2-2.6.9. The result is that every packet is sent to the default queue and I can''t understand why. It seems like iptables is not marking any of the packets, all the queues and classes are empty, traffic always goes through default queues in uplink and downlink. > Here is the script, which is a modification of some things i''ve found in the net: > > #!/bin/bash > # > # > > DEV1=eth1 #salida a red local > DEV0=eth0 #salida a internet > > > # > > TC=/usr/sbin/tc > > if [ "$1" = "status" ] > then > echo "Enlace descendente" > echo "[qdisc]" > $TC -s qdisc show dev $DEV1 > echo "[class]" > $TC -s class show dev $DEV1 > echo "[filter]" > $TC -s filter show dev $DEV1 > > > echo "Enlace ascendente" > echo "[qdisc]" > $TC -s qdisc show dev $DEV0 > echo "[class]" > $TC -s class show dev $DEV0 > echo "[filter]" > $TC -s filter show dev $DEV0 > > # echo "[iptables]" > # iptables -t mangle -L MYSHAPER-OUT -v -x 2> /dev/null > # iptables -t mangle -L MYSHAPER-IN -v -x 2> /dev/null > > > exit > fi > > # Reset everything to a known state (cleared) > $TC qdisc del dev $DEV0 root 2> /dev/null > /dev/null > $TC qdisc del dev $DEV1 root 2> /dev/null > /dev/null > iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-OUT 2> /dev/null > /dev/null > iptables -t mangle -F MYSHAPER-OUT 2> /dev/null > /dev/null > iptables -t mangle -X MYSHAPER-OUT 2> /dev/null > /dev/null > iptables -t mangle -D PREROUTING -i $DEV1 -j MYSHAPER-IN 2> /dev/null > /dev/null > iptables -t mangle -F MYSHAPER-IN 2> /dev/null > /dev/null > iptables -t mangle -X MYSHAPER-IN 2> /dev/null > /dev/null > > #iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-IN 2> /dev/null > /dev/null > > > if [ "$1" = "stop" ] > then > echo "Shaping removed on $DEV1." > echo "Shaping removed on $DEV0." > exit > fi > > ########################################################### > # > # Inbound Shaping (limits total bandwidth to 1000Kbps)If you have 1mbit up and down you need to back off a bit from this (ceils) - upstream to allow for link overheads - how much depending on type of link. Downstream depends on how much you care about latency, as a start say 15-20%, you need to do this to have a queue at all.> # Este es el enlace descendente, desde internet hacia la red interna de Cherrytel > > # set queue size to give latency of about 2 seconds on low-prio packets > ip link set dev $DEV1 qlen 30Makes no difference - if you use sfq you can change a define in the source or use esfq and specify.> > # changes mtu on the outbound device. Lowering the mtu will result > # in lower latency but will also cause slightly lower throughput due > # to IP and TCP protocol overhead. > ip link set dev $DEV1 mtu 1000If I had 1meg symmetrical I doubt I would bother - If you really care that much about latency there are other things to do first. If you do run low MTU I would specify it as quantum for htb and sfq aswell.> > # add HTB root qdisc > $TC qdisc add dev $DEV1 root handle 1: htb default 37 > > # add main rate limit classes > $TC class add dev $DEV1 parent 1: classid 1:1 htb rate 1000kbit > > # add leaf classes - We grant each class at LEAST it''s "fair share" of bandwidth. > # this way no class will ever be starved by another class. Each > # class is also permitted to consume all of the available bandwidth > # if no other classes are in use. > $TC class add dev $DEV1 parent 1:1 classid 1:20 htb rate 64kbit ceil 1000kbit > $TC class add dev $DEV1 parent 1:1 classid 1:21 htb rate 64kbit ceil 1000kbit > $TC class add dev $DEV1 parent 1:1 classid 1:22 htb rate 64kbit ceil 1000kbit > $TC class add dev $DEV1 parent 1:1 classid 1:37 htb rate 832kbit ceil 1000kbit #por defecto > > $TC class add dev $DEV1 parent 1:1 classid 1:23 htb rate 64kbit ceil 64kbit #prueba, maq WiFi > > # attach qdisc to leaf classes - here we at SFQ to each priority class. SFQ insures that > # within each class connections will be treated (almost) fairly. > $TC qdisc add dev $DEV1 parent 1:20 handle 20: sfq perturb 10 > $TC qdisc add dev $DEV1 parent 1:21 handle 21: sfq perturb 10 > $TC qdisc add dev $DEV1 parent 1:22 handle 22: sfq perturb 10 > $TC qdisc add dev $DEV1 parent 1:37 handle 37: sfq perturb 10 > > $TC qdisc add dev $DEV1 parent 1:23 handle 23: sfq perturb 10 > > # filter traffic into classes by fwmark - here we direct traffic into priority class according to > # the fwmark set on the packet (we set fwmark with iptables > # later). Note that above we''ve set the default priority > # class to 1:37 so unmarked packets (or packets marked with > # unfamiliar IDs) will be defaulted to the lowest priority > # class. > $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 20 fw flowid 1:20 > $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 21 fw flowid 1:21 > $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 22 fw flowid 1:22 > $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 23 fw flowid 1:23 > > # Marking the packets. Se marcan los paquetes en el interfaz contrario, para que no se vean > # afectados por el NAT que hacen las reglas del firewall > > iptables -t mangle -N MYSHAPER-OUT > iptables -t mangle -I PREROUTING -i $DEV0 -j MYSHAPER-OUT > > #iptables -t mangle -A MYSHAPER-IN -p tcp --sport ssh -j MARK --set-mark 20 > > > iptables -A MYSHAPER-OUT -d 172.9.264.30 -t mangle -j MARK --set-mark 20 > iptables -A MYSHAPER-OUT -d 172.9.264.31 -t mangle -j MARK --set-mark 20 > iptables -A MYSHAPER-OUT -d 172.9.264.32 -t mangle -j MARK --set-mark 20 > > iptables -A MYSHAPER-OUT -d 172.9.234.22 -t mangle -j MARK --set-mark 21 > iptables -A MYSHAPER-OUT -d 172.9.234.71 -t mangle -j MARK --set-mark 21 > > iptables -A MYSHAPER-OUT -d 172.9.234.25 -t mangle -j MARK --set-mark 22 > > iptables -A MYSHAPER-OUT -d 172.9.234.14 -t mangle -j MARK --set-mark 23 > > # redundant- mark any unmarked packets as 26 (low prio)This won''t mark local adresses as the mangle table in PREROUTING is before de-nat happens. Also I thought 172.x.x.x private range started at 172.16.x.x . You could move MYSHAPER_OUT (though I would call it IN) to FORWARD or use tc filters to match the addresses directly rather than match marks.> > #El resto de tráco irÃal flujo por defecto, el 2:37. > > # Done with inbound shaping > # > #################################################### > > echo "Control del enlace descendente activado." > > #Si solo se desea controlar el enlace descendente, quitar el comentario de la siguiente instruccion exit > #exit > > ########################################################### > # > # Outbound Shaping (limits total bandwidth to 1000Kbps) > # Este es el enlace ascendente, desde la red interna de Cherrytel a internet > > # set queue size to give latency of about 2 seconds on low-prio packets > ip link set dev $DEV0 qlen 30 > > # changes mtu on the outbound device. Lowering the mtu will result > # in lower latency but will also cause slightly lower throughput due > # to IP and TCP protocol overhead. > ip link set dev $DEV0 mtu 1000 > > # add HTB root qdisc > $TC qdisc add dev $DEV0 root handle 2: htb default 73 > > # add main rate limit classes > $TC class add dev $DEV0 parent 2: classid 2:1 htb rate 1000kbit > > # add leaf classes - We grant each class at LEAST it''s "fair share" of bandwidth. > # this way no class will ever be starved by another class. Each > # class is also permitted to consume all of the available bandwidth > # if no other classes are in use. > $TC class add dev $DEV0 parent 2:1 classid 2:70 htb rate 64kbit ceil 1000kbit > $TC class add dev $DEV0 parent 2:1 classid 2:71 htb rate 64kbit ceil 1000kbit > $TC class add dev $DEV0 parent 2:1 classid 2:72 htb rate 64kbit ceil 1000kbit > $TC class add dev $DEV0 parent 2:1 classid 2:87 htb rate 744kbit ceil 1000kbit > > $TC class add dev $DEV0 parent 2:1 classid 2:73 htb rate 64kbit ceil 64kbit #prueba > > # attach qdisc to leaf classes - here we at SFQ to each priority class. SFQ insures that > # within each class connections will be treated (almost) fairly. > $TC qdisc add dev $DEV0 parent 2:70 handle 70: sfq perturb 10 > $TC qdisc add dev $DEV0 parent 2:71 handle 71: sfq perturb 10 > $TC qdisc add dev $DEV0 parent 2:72 handle 72: sfq perturb 10 > $TC qdisc add dev $DEV0 parent 2:87 handle 87: sfq perturb 10 > > $TC qdisc add dev $DEV0 parent 2:73 handle 73: sfq perturb 10 > > # filter traffic into classes by fwmark - here we direct traffic into priority class according to > # the fwmark set on the packet (we set fwmark with iptables > # later). Note that above we''ve set the default priority > # class to 1:87 so unmarked packets (or packets marked with > # unfamiliar IDs) will be defaulted to the lowest priority > # class. > $TC filter add dev $DEV0 parent 2:0 prio 1 protocol ip handle 70 fw flowid 1:70 > $TC filter add dev $DEV0 parent 2:0 prio 2 protocol ip handle 71 fw flowid 1:71 > $TC filter add dev $DEV0 parent 2:0 prio 3 protocol ip handle 72 fw flowid 1:72 > $TC filter add dev $DEV0 parent 2:0 prio 4 protocol ip handle 73 fw flowid 1:73These should be flowid 2:70 not 1:70 etc. Andy.> > # Marking the packets. Se marcan los paquetes en el interfaz contrario, para que no se vean > # afectados por el NAT que hacen las reglas del firewall > > iptables -t mangle -N MYSHAPER-IN > iptables -t mangle -I PREROUTING -i $DEV1 -j MYSHAPER-IN > > #iptables -t mangle -A MYSHAPER-IN -p ! tcp -j MARK --set-mark 20 > > > iptables -A MYSHAPER-IN -s 172.9.234.30 -t mangle -j MARK --set-mark 70 > iptables -A MYSHAPER-IN -s 172.9.234.31 -t mangle -j MARK --set-mark 70 > iptables -A MYSHAPER-IN -s 172.9.234.32 -t mangle -j MARK --set-mark 70 > > iptables -A MYSHAPER-IN -s 172.9.234.22 -t mangle -j MARK --set-mark 71 > iptables -A MYSHAPER-IN -s 172.9.234.71 -t mangle -j MARK --set-mark 71 > > iptables -A MYSHAPER-IN -s 172.9.234.25 -t mangle -j MARK --set-mark 72 > > #Prueba maquina WiFi > iptables -A MYSHAPER-IN -s 172.9.234.14 -t mangle -j MARK --set-mark 73 > > #El resto de tráco irÃal flujo por defecto, el 2:87. > > > # Done with outbound shaping > > #################################################### > > echo "Control del enlace ascendente activado." > > exit > > Thanks for your help! > > > > UN CORDIAL SALUDO > > Miguel Ángel Domínguez Durán. > Departamento Técnico. > Cherrytel Comunicaciones, S.L. > mdominguez@cherrytel.com > http://www.cherrytel.com/ > Tlf. 902 115 673 > Fax 952218170_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/