LARTC - Sep 2007 - Classes do not receive any traffic ?

Hello everyone.


CONFIGURATION DESCRIPTION:
I have a linux box doing masquerade for two lan''s. Here is a piece of 
mine network config:

eth0 : ISP , one public ip address (DSL modem)
eth1 : lan , private network address fe: 192.168.4.0/24
eth2 : wlan access point performing as lan2wlan bridge , private network 
addes fe. 192.168.67.0/24

This box use 2.6.20 kernel with iptables-1.3.8 and 
iproute-2.6.20-070313. I also use imq hooked as AB.



I have tried to shape eth0''s incoming traffic using imq0 :

===============================start===================================
ifconfig imq0 txqueuelen 30 up

tc qdisc add dev imq0 root handle 1: htb ${root_parm}

#this class have speed corresponding to maximum achieved speed of isp 
connection in particular direction
tc class add dev imq0 parent 1:0 classid 1:1 htb rate ${rate_sum}
ceil ${rate_sum} burst 0kb cburst 0kb ${quantum_sum}

# summarized prio class
tc class add dev imq0 parent 1:1 classid 1:2 htb rate ${rate_prio_sum} 
ceil ${rate_prio_sum} burst 0kb cburst 0kb ${quantum_prio_sum}

#critical prio
tc class add dev imq0 parent 1:2 classid 1:3 htb prio ${prio_crit} rate 
${rate_crit} ceil ${ceil_crit} burst 0kb cburst 0kb ${quantum_crit}
tc filter add dev imq0 parent 1:0 protocol ip prio 0 u32 match mark 3 
0xffff flowid 1:3

#prio voip
tc class add dev imq0 parent 1:2 classid 1:4 htb prio ${prio_voip} rate 
${rate_voip} ceil ${ceil_voip} burst 0kb cburst 0kb ${quantum_voip}
tc filter add dev imq0 parent 1:0 protocol ip prio 1 u32 match mark 4 
0xffff flowid 1:4

#games
tc class add dev imq0 parent 1:2 classid 1:5 htb prio ${prio_ent} rate 
${rate_ent} ceil ${ceil_ent} burst 0kb cburst 0kb ${quantum_ent}
tc filter add dev imq0 parent 1:0 protocol ip prio 2 u32 match mark 5 
0xffff flowid 1:5

# summarized class for generic unclassified traffic
tc class add dev imq0 parent 1:1 classid 1:6 htb rate ${rate_user_sum} 
ceil ${rate_user_sum} burst 0kb cburst 0kb ${quantum_user_sum}

#generic server traffic
tc class add dev imq0 parent 1:6 classid 1:7 htb prio ${prio_srv} rate 
${rate_srv} ceil ${ceil_srv} burst 0kb cburst 0kb ${quantum_srv}
tc filter add dev imq0 parent 1:0 protocol ip prio 4 u32 match ip 
${dev[3]} $server_public_ip flowid 1:7

#default class
tc class add dev imq0 parent 1:6 classid 1:8 htb prio 7 rate 1kbps ceil 
1kbps burst 0kb cburst 0kb quantum 1500

# fe. single user class
tc class add dev imq0 parent 1:6 classid 1:30 htb prio ${prio_user} rate 
${rate_user} ceil ${ceil_user} burst 0kb cburst 0kb ${quantum_user}
tc filter add dev imq0 parent 1:0 protocol ip prio 5 u32 match ip 
${dev[3]} 192.168.4.5 flowid 1:30
					
#Each user has his own class for traffic that were not classified for 
#prio classes. Other traffic is probably traffic from "unofficial users 
#so they receive what they deserve :).

#This two functions are responsible for setting up iptables and imq hooking.

skype() 
 

{
#Nasty workaround for skype 
 

if [ ! -r /tmp/1 -o ! -r /tmp/2 ] ; then	
touch /tmp/1 /tmp/2 
 

find /etc/l7-protocols/protocols/ -type f -name *pat |cut -d/ -f5 | cut 
-d. -f1 >/tmp/1 

sed 
''/skype*/d;/finger*/d;/biff*/d;/whois*/d;/tsp*/d;/ntp*/d;/unknown*/d;/sip*/d;/h232*/d;/teamspeak*/d;/ventrilo*/d;/ssh*/d;/jabber*/d;/aim*/d;/msnmessenger*/d;/yahoo*/d;/qq*/d;/battlefield1942/d;/battlefield2/d;/counterstrike-source*/d;/dayofdefeat-source*/d;/doom3*/d;/halflife2-deathmatch*/d;/mohaa*/d;/quake-halflife*/d;/quake1*/d;/worldofwarcraft*/d''
/tmp/1 > /tmp/2
fi
i=`cat /tmp/2|wc -l` 
 

j=1 
 

iptables -t mangle -N ${dev[2]}_SKYPE 
 

iptables -t mangle -A ${dev[2]}_SKYPE -p tcp --sport 1:1024 -j RETURN 
 

iptables -t mangle -A ${dev[2]}_SKYPE -p udp --sport 1:1024 -j RETURN 
 

iptables -t mangle -A ${dev[2]}_SKYPE -p tcp --dport 1:1024 -j RETURN 
 

iptables -t mangle -A ${dev[2]}_SKYPE -p udp --dport 1:1024 -j RETURN 
 

while [ ${j} -le ${i} ]; do
iptables -t mangle -A ${dev[2]}_SKYPE -m layer7 --l7proto `sed -n ${j}p 
/tmp/2` -j RETURN 

j=$(($j+1)) 
 

done
iptables -t mangle -A ${dev[2]}_SKYPE -m layer7 --l7proto skypetoskype 
-j ${dev[2]}_CON_VOIP
iptables -t mangle -A ${dev[2]}_SKYPE -m layer7 --l7proto skypeout -j 
${dev[2]}_CON_VOIP>/dev/null 1>/dev/null 2>/dev/null 3>/dev/null
4>/dev/null
iptables -t mangle -A ${dev[2]}_SKYPE -j RETURN 

}

ipt_int()
{
iptables -t mangle -N ${dev[2]}_CHECK

iptables -t mangle -N ${dev[2]}_IMQ


iptables -t mangle -N ${dev[2]}_PRIO
if [[ ${dev[0]} =~ ''imq0'' ]] ; then
iptables -t mangle -A ${dev[2]}_PRIO -j MARK --set-mark 3
elif [[ ${dev[0]} =~ ''eth0'' ]] ; then
iptables -t mangle -A ${dev[2]}_PRIO -j CLASSIFY --set-class 1:3
else
exit 1
fi
iptables -t mangle -A ${dev[2]}_PRIO -j ${dev[2]}_IMQ

iptables -t mangle -N ${dev[2]}_CON_PRIO
if [[ ${dev[0]} =~ ''imq0'' ]] ; then
iptables -t mangle -A ${dev[2]}_CON_PRIO -j MARK --set-mark 3
elif [[ ${dev[0]} =~ ''eth0'' ]] ; then
iptables -t mangle -A ${dev[2]}_CON_PRIO -j CLASSIFY --set-class 1:3
else
exit 1
fi
iptables -t mangle -A ${dev[2]}_CON_PRIO -j CONNMARK --save-mark
iptables -t mangle -A ${dev[2]}_CON_PRIO -j ${dev[2]}_IMQ

iptables -t mangle -N ${dev[2]}_CON_VOIP
if [[ ${dev[0]} =~ ''imq0'' ]] ; then
iptables -t mangle -A ${dev[2]}_CON_VOIP -j MARK --set-mark 4
elif [[ ${dev[0]} =~ ''eth0'' ]] ; then
iptables -t mangle -A ${dev[2]}_CON_VOIP -j CLASSIFY --set-class 1:4
else
echo coś nie tak w ''ip_int()''
exit 1
fi
iptables -t mangle -A ${dev[2]}_CON_VOIP -j CONNMARK --save-mark
iptables -t mangle -A ${dev[2]}_CON_VOIP -j ${dev[2]}_IMQ

iptables -t mangle -N ${dev[2]}_CON_GRY
if [[ ${dev[0]} =~ ''imq0'' ]] ; then
iptables -t mangle -A ${dev[2]}_CON_GRY -j MARK --set-mark 5
elif [[ ${dev[0]} =~ ''eth0'' ]] ; then
iptables -t mangle -A ${dev[2]}_CON_GRY -j CLASSIFY --set-class 1:5
else
exit 1
fi
iptables -t mangle -A ${dev[2]}_CON_GRY -j CONNMARK --save-mark
iptables -t mangle -A ${dev[2]}_CON_GRY -j ${dev[2]}_IMQ

iptables -t mangle -N ${dev[2]}_GRY
if [[ ${dev[0]} =~ ''imq0'' ]] ; then
iptables -t mangle -A ${dev[2]}_GRY -j MARK --set-mark 5
elif [[ ${dev[0]} =~ ''eth0'' ]] ; then
iptables -t mangle -A ${dev[2]}_GRY -j CLASSIFY --set-class 1:5
else
exit 1
fi
iptables -t mangle -A ${dev[2]}_GRY -j ${dev[2]}_IMQ


iptables -t mangle -A ${dev[2]}_CHECK -m length --length 1:64 -p tcp 
--tcp-flags SYN,FIN,ACK ACK -j ${dev[2]}_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -p tcp --tcp-flags SYN,ACK,FIN SYN 
-j ${dev[2]}_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -p tcp --tcp-flags SYN,ACK,FIN 
SYN,ACK -j ${dev[2]}_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -p tcp --tcp-flags SYN,ACK,FIN 
FIN,ACK -j ${dev[2]}_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -p icmp --icmp-type echo-request 
-j ${dev[2]}_GRY
iptables -t mangle -A ${dev[2]}_CHECK -p icmp --icmp-type echo-reply -j 
${dev[2]}_GRY
iptables -t mangle -A ${dev[2]}_CHECK -j CONNMARK --restore-mark
iptables -t mangle -A ${dev[2]}_CHECK -m mark ! --mark 0 -j ${dev[2]}_IMQ
iptables -t mangle -A ${dev[2]}_CHECK -p tcp -m multiport --ports 
22,53,65522 -j ${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -p udp -m multiport --ports 
22,53,65522 -j ${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -p tcp -m iprange 
--${dev[4]}-range 217.17.41.80-217.17.41.95 -m multiport 
--${dev[6]}ports 8074,443 -j ${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -p tcp -m iprange 
--${dev[4]}-range 217.17.45.128-217.17.45.159 -m multiport 
--${dev[6]}ports 8074,443 -j ${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto ssh -j 
${dev[2]}_CON_PRIO

iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto sip -j 
${dev[2]}_CON_VOIP
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto h323 -j 
${dev[2]}_CON_VOIP
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto teamspeak -j 
${dev[2]}_CON_VOIP
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto ventrilo -j 
${dev[2]}_CON_VOIP
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto jabber -j 
${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto aim -j 
${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto msnmessenger 
-j ${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto yahoo -j 
${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto qq -j 
${dev[2]}_CON_PRIO
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto 
battlefield1942 -j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto battlefield2 
-j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto 
counterstrike-source -j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto 
dayofdefeat-source -j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto doom3 -j 
${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto 
halflife2-deathmatch -j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto mohaa -j 
${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto quake-halflife 
-j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto quake1 -j 
${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -m layer7 --l7proto 
worldofwarcraft -j ${dev[2]}_CON_GRY
	
skype
	
iptables -t mangle -A ${dev[2]}_CHECK -j ${dev[2]}_SKYPE
iptables -t mangle -A ${dev[2]}_CHECK -p tcp -m multiport --ports 
27000:27030,28960 -j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -p udp -m multiport --ports 
27000:27030,27901,27960,28960,28960,28960,14567,16567 -j ${dev[2]}_CON_GRY
iptables -t mangle -A ${dev[2]}_CHECK -j ${dev[2]}_IMQ
	
iptables -t mangle -N ${dev[2]}_KONIEC
iptables -t mangle -A ${dev[2]}_KONIEC -j RETURN
	
if [[ ${dev[0]} =~ ''imq(.*)'' ]] ; then
export numdev=${BASH_REMATCH[1]}
iptables -t mangle -A ${dev[2]}_IMQ -j IMQ --todev ${numdev}
fi
iptables -t mangle -A ${dev[2]}_IMQ -j ${dev[2]}_KONIEC
if [[ ${dev[2]} =~ ''DL'' ]] ; then
iptables -t mangle -I ${dev[7]} -i ${dev[1]} -j ${dev[2]}_CHECK
elif [[ ${dev[2]} =~ ''UL'' ]] ; then
iptables -t mangle -I ${dev[7]} -o ${dev[1]} -j ${dev[2]}_CHECK
else
exit 1
fi
}

# Before invoking any function i setup correct data to dev vector:

#example
export dev=(imq0 eth0 DL dst src d s PREROUTING)

===============================stop==================================


I know that my script is really complex one. Check this simplified 
diagram to understand general idea:


-----|
P    |
R    |
E   R|
     O|		           /=YES=>[prio]=><imq/end>
O   U|		          ||
R   T|=>(?mark!=0?)=NO=>(?prio?)=NO=>(?valid
user?)=NO=>[def]=><imq/end>
     I|	    ||				||
P   N|	     \=YES=><imq/end>		 \=YES=>[user]=><imq/end>
O   G|
S    |
T    |
-----|

[] - classifying for proper class (mark or u32)
<imq/end> - packet travels to imq or leaves PRE,POSTROUTING




QUESTIONS AND PROBLEM DESCRIPTION:

1.Main problem.
It seems that classes on imq0 that should shape incoming traffic from 
internet do not recognizes marks. Fw match don''t work. U32 match works 
except matching marks. The only classes that receive traffic on imq0 are 
server class and user classes. Similar problem occurred on eth0(upload) 
but I managed to solve this problem by using -j CLASSIFY instead -j 
MARK. When I tried to fix this problem I have learned that this may be 
caused by the way tc and iptables are works together.I am sure that 
marks are set and IMQ target works (non zero iptables/ifconfig counters) 
. I think that it is possible for u32 matches to classify traffic before 
any mark is set. Unfortunately kptd is out of date so it is not certain 
to me. Would somebody explain me why fwmark do not work on imq0 ?

2.
I have found that when i try to ping from host in lan to host in 
internet every fifth icmp packet has significantly higher delay. F.e. 
four packets goes trough with delay approx 15ms but next packet have 
delay up to 100ms ! I suppose that it may be caused by to big txqueuelen 
so i decreased it from 1000 to 30 on all interfaces without any problems 
with lesser bandwidth or packet looses. Could somebody advice proper 
value for txqueuelen if it was a good idea to change it.
I have 1Mbit/256kbit DSL modem.

3.
Is it a good idea to set proper ToS value for a outbound traffic that 
was classified as prio ?? Would it give any decrease in delays ??




I hope I will find someone helpful and also very tolerant for mine poor 
English ... Posting on LTARC is the only way to solve mine problems....


Bartek





----------------------------------------------------------------------
Bedac w WC czytala wiadomosci.
>>> http://link.interia.pl/f1b71

bartekR wrote:
> 1.Main problem.
> It seems that classes on imq0 that should shape incoming traffic from 
> internet do not recognizes marks. Fw match don''t work. U32 match
works
> except matching marks. The only classes that receive traffic on imq0 are 
> server class and user classes. Similar problem occurred on eth0(upload) 
> but I managed to solve this problem by using -j CLASSIFY instead -j 
> MARK. When I tried to fix this problem I have learned that this may be 
> caused by the way tc and iptables are works together.I am sure that 
> marks are set and IMQ target works (non zero iptables/ifconfig counters) 
> . I think that it is possible for u32 matches to classify traffic before 
> any mark is set. Unfortunately kptd is out of date so it is not certain 
> to me. Would somebody explain me why fwmark do not work on imq0 ?
Mark should work on IMQ if set in PREROUTING mangle - so should 
classify, so perhaps you could work around with that.

You say it didn''t work on eth0 either so maybe your/your
distro''s kernel
config is to blame here''s what I get greping for mark/fw

andy@noki:~$ grep -i mark /boot/config-2.6.21.1
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_CLS_U32_MARK=y
andy@noki:~$ grep -i fw /boot/config-2.6.21.1
CONFIG_NET_CLS_FW=m
CONFIG_FW_LOADER=m

If your config looks OK your rules are so complex I would try something 
very simple like marking icmp and making a filter for that on eth/imq 
and seeing if it works or not.

iptables counters don''t tell you if you later accidently cleared the 
mark and -j IMQ won''t jump to IMQ from that place in the script.
> 
> 2.
> I have found that when i try to ping from host in lan to host in 
> internet every fifth icmp packet has significantly higher delay. F.e. 
> four packets goes trough with delay approx 15ms but next packet have 
> delay up to 100ms ! I suppose that it may be caused by to big txqueuelen 
> so i decreased it from 1000 to 30 on all interfaces without any problems 
> with lesser bandwidth or packet looses. Could somebody advice proper 
> value for txqueuelen if it was a good idea to change it.
> I have 1Mbit/256kbit DSL modem.
DSl rates are hard to get right without patching tc/kernel as it uses 
ATM and the overheads on a packet are high and vary with size in 53byte 
chunks. Without patching you need to back off from the rates.

To make things worse ingress shaping needs you to back off even more as 
you are at the wrong end of the bottleneck, so don''t have total control
like on egress. The slower the link the harder it is, you should use 
short child qdiscs on the htb bulk classes (limit parameter) and try to 
arrange the htb rules so that interactive classes (and don''t have too 
many) get a rate much higher than they ever need with a higher prio than 
bulk.

For htb prio 0 is top - for tc filters it''s 1
> 
> 3.
> Is it a good idea to set proper ToS value for a outbound traffic that 
> was classified as prio ?? Would it give any decrease in delays ??
I don''t think it will make any difference.

Andy.

Andy Furniss wrote:
> If your config looks OK your rules are so complex I would try something 
> very simple like marking icmp and making a filter for that on eth/imq 
> and seeing if it works or not.
I have built simple script and as You wrote it worked. After this I have 
modified mine script by:

-removing lines containing -j CONNMARK and line matching non zero mark.
-changing a few returning lines in _SKYPE

Now every classes receive traffic as it should and ping decreased 
significantly.


Andy Furniss wrote:
> DSl rates are hard to get right without patching tc/kernel as it uses 
> ATM and the overheads on a packet are high and vary with size in 53byte 
> chunks. Without patching you need to back off from the rates.
I didn''t knew that. Would You give me larger information about this ??

Andy Furniss wrote:
> To make things worse ingress shaping needs you to back off even more as 
> you are at the wrong end of the bottleneck, so don''t have total
control
> like on egress. The slower the link the harder it is, you should use 
> short child qdiscs on the htb bulk classes (limit parameter) and try to 
> arrange the htb rules so that interactive classes (and don''t have
too
> many) get a rate much higher than they ever need with a higher prio than 
> bulk.
I will try things as You mentioned.

I have 3 interactive classes because of Skype and p2p. I had tried to 
match Skype file transfers to user classes. Unfortunately Skype 
connections are encrypted. Mine first idea was to mark this packet by 
matching tos (similar to matching tos of ssh and scp) but it failed 
because Skype always send packet with tos=0. Is there any other way to 
do that ??
Sometimes p2p puts lots of ack packets. I don''t want them to interfere 
with low latencies needed for games.



Thank You for Your help.
Especially I appreciate Your will to read and understand mine first 
,very long and complex post :)


Bartek


----------------------------------------------------------------------
Bedac w WC czytala wiadomosci.
>>> http://link.interia.pl/f1b9c

bartekR wrote:
> Andy Furniss wrote:
>> DSl rates are hard to get right without patching tc/kernel as it uses 
>> ATM and the overheads on a packet are high and vary with size in 
>> 53byte chunks. Without patching you need to back off from the rates.
> 
> I didn''t knew that. Would You give me larger information about
this ??
There may be a way to allow for them in kernel one day, but for now you 
need to patch htb and tc to do it.

http://ace-host.stuart.id.au/russell/files/tc/tc-atm/

There is also a patch to make htb more accurate on that page.

Even if you patch you still have to back of for ingress traffic or it 
will buffer up at your ISP/Teleco and mess up latency. The more you care 
about latency the more you need to sacrifice bandwidth.

On 256kbit egress you are going to get latency up to 50ms because
that''s
how long a 1500 byte packet will take to transmit. If you don''t tweak 
htb it will be up to 100 because htb dequeues in pairs by default.

If you want less you could consider lowering mtu or mss clamping.
> 
> Andy Furniss wrote:
>> To make things worse ingress shaping needs you to back off even more 
>> as you are at the wrong end of the bottleneck, so don''t have
total
>> control like on egress. The slower the link the harder it is, you 
>> should use short child qdiscs on the htb bulk classes (limit 
>> parameter) and try to arrange the htb rules so that interactive 
>> classes (and don''t have too many) get a rate much higher than
they
>> ever need with a higher prio than bulk.
> 
> I will try things as You mentioned.
> 
> I have 3 interactive classes because of Skype and p2p. I had tried to 
> match Skype file transfers to user classes. Unfortunately Skype 
> connections are encrypted. Mine first idea was to mark this packet by 
> matching tos (similar to matching tos of ssh and scp) but it failed 
> because Skype always send packet with tos=0. Is there any other way to 
> do that ??
> Sometimes p2p puts lots of ack packets. I don''t want them to
interfere
> with low latencies needed for games.
Sometimes it''s easier to match what you know is interactive and treat 
the rest as bulk, you can still give acks/small packets a higher prio 
than the rest, but put known game traffic highest prio.

I noticed you used 0 burst - IIRC 0 ended up bigger than using 10 for me 
- I guess you get a default if you specify 0. You should only do this 
for bulk traffic, for interactive it''s better to let it burst through. 
You don''t really want to delay it at all.