LARTC - Mar 2006 - Please help - totally confused (NAT + FWMARK + IMQ + HTB)

Hello,
 
I''m trying to get my shaper to work, but have only a partial success.
Can someone help me with that. My setup unfortunately is not so trivial,
but I think some people could have similar one...
 
1. There is a router connected to the internet line via interface eth0
2. There are users connected to the router via two interfaces : eth1 and
wlan0
3. All users are assigned private IP addresses (192.168.1.xxx on eth1,
192.168.2.xxx on wlan0)
4. The number of public IP addresses is limited, so the router does SNAT
(and for some users having assigned a public IP address also DNAT)
5. For the traffic classification I need to use iptables (and MARK
target)
6. For the traffic shaping, I need to use HTB
7. Each user has only one IP address and should have allocated some
upload and download bandwitdh
8. I need to get both UPLOAD and DOWNLOAD shaping, based on user''s
private address
 
Please can someone post me some *really working* script for that? Or at
least tell me, where is my fault? I think it could be in sequence of
iptables calls, POSTROUTING/PREROUTING misunderstanding, etc...
 
Thank you. 
Jan
 
Here is my script:
 
#clean everything
 
iptables -X
iptables -X -t nat
iptables -X -t mangle
 
iptables -F
iptables -F -t nat
iptables -F -t mangle

# NAT setup
 
# user 1 nat
iptables -t nat -A POSTROUTING -s $user1ip -j SNAT --to-source
$publicip1
 
(repeat for all users)
 
# setup IMQ
ip link set imq0 up
ip link set imq1 up
 
# flush mangle
iptables -t mangle -F
 
# MARK user1''s traffic - mark normal and priority traffic
 
# mark download(?)(postrouting?)
iptables -t mangle -A POSTROUTING -d $user1ip -j MARK --set-mark 0x1890
iptables -t mangle -A POSTROUTING -d $user1ip (and other options for
priority download) -j MARK --set-mark 0x2890
 
# mark upload(?) (prerouting?)
iptables -t mangle -A PREROUTING -s $user1ip -j MARK --set-mark 0x3890
iptables -t mangle -A PREROUTING -s $user1ip (and other options for
priority upload) -j MARK --set-mark 0x4890
(repeat for all users)

# Finally, jump to IMQ
 
# imq0 - download (???) (do I need --to-dev?) (are -o options OK?)
iptables -t mangle -A POSTROUTING -o wlan0 -j IMQ --todev 0
iptables -t mangle -A POSTROUTING -o eth1 -j IMQ --todev 0
 
# imq0 - download (???) (do I need --to-dev?) (are -i options OK?)
iptables -t mangle -A PREROUTING -i wlan0 -j IMQ --todev 1
iptables -t mangle -A PREROUTING -i eth1 -j IMQ --todev 1

# DOWNLOAD (?) SHAPER
tc qdisc del dev imq0 root
tc qdisc add dev imq0 root handle 1: htb default 0x9999
tc class add dev imq0 parent 1: classid 1:1 htb rate 3850kbit
tc class add dev imq0 parent 1:1 classid 1:0x9999 htb rate 128kbit ceil
3850kbit prio 3
 
# UPLOAD (?) SHAPER
tc qdisc del dev imq1 root
tc qdisc add dev imq1 root handle 2: htb default 0x9999
tc class add dev imq1 parent 2: classid 2:1 htb rate 3850kbit
tc class add dev imq1 parent 2:1 classid 2:0x9999 htb rate 128kbit ceil
3850kbit prio 3

# DEFINE USER''S class (main class 0x890, default class 0x1890, priority
class 0x2890)
 
# download (?)
tc class add dev imq0 parent 1:1 classid 1:0x890 htb rate 128kbit ceil
1024kbit prio 3
tc class add dev imq0 parent 1:0x890 classid 1:0x1890 htb rate 128kbit
ceil 1024kbit prio 3
tc class add dev imq0 parent 1:0x890 classid 1:0x2890 htb rate 128kbit
ceil 1024kbit prio 1
tc qdisc add dev imq0 parent 1:0x1890 handle 0x1890: sfq perturb 10
tc qdisc add dev imq0 parent 1:0x2890 handle 0x2890: sfq perturb 10
tc filter add dev imq0 parent 1:0 protocol ip handle 0x1890 fw flowid
1:0x1890
tc filter add dev imq0 parent 1:0 protocol ip handle 0x2890 fw flowid
1:0x2890
 
# upload (?)
tc class add dev imq1 parent 2:1 classid 2:0x890 htb rate 128kbit ceil
1024kbit prio 3
tc class add dev imq1 parent 2:0x890 classid 2:0x1890 htb rate 128kbit
ceil 1024kbit prio 3
tc class add dev imq1 parent 2:0x890 classid 2:0x2890 htb rate 128kbit
ceil 1024kbit prio 1
tc qdisc add dev imq1 parent 2:0x1890 handle 0x1890: sfq perturb 10
tc qdisc add dev imq1 parent 2:0x2890 handle 0x2890: sfq perturb 10
tc filter add dev imq1 parent 2:0 protocol ip handle 0x3890 fw flowid
1:0x1890
tc filter add dev imq1 parent 2:0 protocol ip handle 0x4890 fw flowid
1:0x2890

(repeat for each user)


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

On Tue, Mar 28, 2006 at 10:07:36AM +0200, Jan Rovner
wrote:
> 1. There is a router connected to the internet line via interface eth0
That''s fine.
> 2. There are users connected to the router via two interfaces : eth1 and
> wlan0
Two possibilities come to mind:

a) If you can provide two completely separate bandwidth pools,
   you can use one HTB qdisc per device.
b) Otherwise you have to use a virtual device, for example IMQ.
> 3. All users are assigned private IP addresses (192.168.1.xxx on eth1,
> 192.168.2.xxx on wlan0)
That''s fine.
> 4. The number of public IP addresses is limited, so the router does SNAT
> (and for some users having assigned a public IP address also DNAT)
More than one public IP address, but only one physical line, right?
That seems to be fine.
> 5. For the traffic classification I need to use iptables (and MARK
> target)
> 6. For the traffic shaping, I need to use HTB
> 7. Each user has only one IP address and should have allocated some
> upload and download bandwitdh
> 8. I need to get both UPLOAD and DOWNLOAD shaping, based on user''s
> private address
Alright, judging from your description, it should be possible to do 
things that way.
> Please can someone post me some *really working* script for that? Or at
> least tell me, where is my fault? I think it could be in sequence of
> iptables calls, POSTROUTING/PREROUTING misunderstanding, etc...
I don''t have a working script for exactly that; mine uses just one 
interface on the download side and only one public IP. But it distributes 
bandwidth on a per-user basis using HTB. I''ve also put some effort into
documenting it, so maybe it can serve as an example:
http://www.metamorpher.de/fairnat/
> # setup IMQ
> ip link set imq0 up
> ip link set imq1 up
Since I''m not using IMQ myself, I''m not sure about this part, 
but why are you using two devices? imq0 seems fine, but imq1 
looks wrong to me. I would do the upload shaping on your 
internet device (eth0) directly.

About your script, depending on what is working and what is 
not, you can debug it by doing the following:

- verify that the iptables rules match the packets you want
  it to match. For example, iptables can list you the rules 
  it is using as well as counters for them. Or you could 
  add some logging rules. If the packets are not matched, 
  and thus not getting marked, your shaping can not work.

- verify that the packets go in the HTB classes you want
  them to go. This can probably be done by using HTB 
  statistics (tc -s -d qdisc/class show ...).

If you can describe in more detail what is (not) working 
about your script, maybe I can give you some better hints. 
Just by glancing at a script without knowing what is wrong 
it''s hard to give recommendations.

Regards
Andreas Klauer