LARTC - Mar 2003 - Routing/forwarding/shaping problems in v2.2.x (Long - sorry)

Greetings from a newbie!
(Well, to this list anyway)

I''m having a problem and I hope someone here might be able to help...

I am strongly expecting an answer along the lines of "upgrade to
v2.4.x", but
I would REALLY preffer to avoid that for now...

The setup:

"Home brewed" v2.2.24 (will patch to v2.2.25 later today) with the DS8
patch
applied. Currently downloading the DS9/rbtree/htb3 patches to be applied 
later (obviously, unpatching the old DS8 first), and see if at least some of 
my problems go away.

Multiple cable/DSL lines with multiple default routes and equal cost 
multipath.

The problems:

1) Ingress shaping/policing doesn''t seem to work at all.

I haven''t tried outgoing shaping/policing, but prioritizing traffic
definitely
works, and since activating it, my bandwidth usage has quite obviously jumped 
to closer to my available limits (from the bandwidth usage graphs) on 
cable/DSL connections. ssh and ping latency has also gone through the floor 
after increasing the priority for those protocols (using a variant of the 
wonderhaper). So that clearly seems to work.

When applying ingres shaping (policing filter) all executes fine without 
reporting any errors, but

tc -s -d qdisc show dev eth1
and
tc -s -d filter show dev eth1

both say that no traffic has been caught by the rules (which just cannot be 
right because I am using a u32 filter with src 0.0.0.0/0, and the same filter 
works for outgoing traffic with dst 0.0.0.0/0.

Here is a script snippet:

tc qdisc add    dev $DEV ingress        \
                handle ffff:

tc filter add   dev $DEV                        \
                parent ffff:                    \
                protocol ip                     \
                prio 50                         \
                u32 match ip                    \
                src 0.0.0.0/0                   \
                police rate ${DOWNLINK}Kbit     \
                burst $[8*DOWNLINK]Kbit         \
                drop flowid :1


Can anyone hazard a guess as to why this is not doing what it should be? Is 
this a know bug in DS8 and DS9 will fix it? I will try it anyway, just to 
make sure, but some encouraging news would be nice. :-)

1.2) If the ingress traffic shaping is unfixable in v2.2.x, would it be 
possible to instead set up a dummy network device, and set up an egress 
shaper on the physical interface, and forward everything to the dummy 
intgerface, and then use the dummy interface as the default gateway? 
Effectively this would do the same thing as setting up two routers 
bacl-to-back, and using only egress shaping on both routers to achieve 
ingress shaping? Kind of like having a logical, rather than a physical 
router?

2) ipmasqadm portfw unstable/unreliable

I have tried to use this approach to forward ports from the firewall to an 
internal server. It works OK initially, but within minutes, things start 
going wrong. Some connections get through on one interface but not the other. 
Later, connections from the same host will work on a different interface, but 
not the one it worked on initially.

This can be temporarily made to go away by doing
# ip route flush cache
a few times, but the problem will always return. After about a month of 
uptime, this will get to the point where most connections fail on most 
interfaces, and the only cure I have found is a reboot.

Connections that are actually physically proxied by using NetCat (where the 
source address doesn''t need to be preserved) or redir (with
--transproxy
where the source address does need to be preserved) work fine, and
don''t
experience the same problem at all.

Obviouly, I would prefer to use the "ipmasqadm portfw" solution
because it is
massively more memory/CPU efficient than a generic user-space tcp proxy 
deamon running from inetd.

3) teql vs. multiple default routes
Not really a problem, more of a matter of exploring options, and I
haven''t
found any particularly detailed information on the subject.

Can a virtual load ballancing device be used to aggregate multiple ethernet 
interfaces? Presumably load balancing of incoming things is still done in the 
traditional way (DNS+stuff), and a teql interface would only be used to load 
balance connections initiated from the network itself?

Presumably, the advantage of using a teql interface over multiple default 
routes is that load ballancing would be done per session, rather than per 
route? Are there any routing issues from the external point of view with 
using teql, or is it completely transparent to the outside world? What 
concerns me is that the source IP address might be a private IP address of 
the teql interface. Are there any potential issues with this, or can standard 
ipchains/masquerading rules be used to overcome any potential problems?

TIA.

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

On Thu, 20 Mar 2003, Gordan Bobic wrote:
> The setup:
>
> "Home brewed" v2.2.24 (will patch to v2.2.25 later today) with
the DS8 patch
> applied. Currently downloading the DS9/rbtree/htb3 patches to be applied
> later (obviously, unpatching the old DS8 first), and see if at least some
of
> my problems go away.
	Yes, DS-8 has many problems including security ones.
The only problem is that I still didn''t upgraded the patches to
2.2.25, may be in the next days I''ll find time to do so.
> Multiple cable/DSL lines with multiple default routes and equal cost
> multipath.
	I strongly recommend the route patches in such case:

http://www.ssi.bg/~ja/#routes-2.2
http://www.ssi.bg/~ja/routes-2.2.20-7.diff
> When applying ingres shaping (policing filter) all executes fine without
> reporting any errors, but
>
> tc -s -d qdisc show dev eth1
> and
> tc -s -d filter show dev eth1
	DS9 has fixes for the ingress stats
> Can anyone hazard a guess as to why this is not doing what it should be? Is
> this a know bug in DS8 and DS9 will fix it? I will try it anyway, just to
> make sure, but some encouraging news would be nice. :-)
	Yes, yes, I''ll add it to the changelog
> 2) ipmasqadm portfw unstable/unreliable
>
> I have tried to use this approach to forward ports from the firewall to an
> internal server. It works OK initially, but within minutes, things start
> going wrong. Some connections get through on one interface but not the
other.
> Later, connections from the same host will work on a different interface,
but
> not the one it worked on initially.
	Such problems should be solved from the "routes" patches,
they will keep each traffic through its ISP.

	Read nano.txt from http://www.ssi.bg/~ja/#routes
It is for 2.4 but the concept and the routing rules are same.
> TIA.
>
> Gordan
Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Thursday 20 Mar 2003 12:59, Julian Anastasov wrote:
> Hello,
>
> On Thu, 20 Mar 2003, Gordan Bobic wrote:
> > The setup:
> >
> > "Home brewed" v2.2.24 (will patch to v2.2.25 later today)
with the DS8
> > patch applied. Currently downloading the DS9/rbtree/htb3 patches to be
> > applied later (obviously, unpatching the old DS8 first), and see if at
> > least some of my problems go away.
>
> 	Yes, DS-8 has many problems including security ones.
> The only problem is that I still didn''t upgraded the patches to
> 2.2.25, may be in the next days I''ll find time to do so.
It all patched cleanly anyway, don''t worry about it. I patched 2.2.24
and then
patched that to 2.2.25. I haven''t compiled it yet because I am not in
front
of the machine right now (not doing remote kernel upgrades - bad things 
happen). :-)
> > Multiple cable/DSL lines with multiple default routes and equal cost
> > multipath.
>
> 	I strongly recommend the route patches in such case:
>
> http://www.ssi.bg/~ja/#routes-2.2
> http://www.ssi.bg/~ja/routes-2.2.20-7.diff
Aha! Thanks. I wonder if whis will cure my problems. BTW, that applies cleanly 
to my already heavily patched 2.2.25 tree.
> > When applying ingres shaping (policing filter) all executes fine
without
> > reporting any errors, but
> >
> > tc -s -d qdisc show dev eth1
> > and
> > tc -s -d filter show dev eth1
>
> 	DS9 has fixes for the ingress stats
Marvellous. :-)
> > Can anyone hazard a guess as to why this is not doing what it should
be?
> > Is this a know bug in DS8 and DS9 will fix it? I will try it anyway,
just
> > to make sure, but some encouraging news would be nice. :-)
>
> 	Yes, yes, I''ll add it to the changelog
:-)
> > 2) ipmasqadm portfw unstable/unreliable
> >
> > I have tried to use this approach to forward ports from the firewall
to
> > an internal server. It works OK initially, but within minutes, things
> > start going wrong. Some connections get through on one interface but
not
> > the other. Later, connections from the same host will work on a
different
> > interface, but not the one it worked on initially.
>
> 	Such problems should be solved from the "routes" patches,
> they will keep each traffic through its ISP.
Excellent. Note that I also use policy routing so that the sessions started to 
ethX will always be reponded to on ethX. I think that is fairly standard 
(things break horribly otherwise). However, what happens when a host randomly 
alternates between IP addresses it is starting sessions to/from? This sort of 
worked before, but it quickly started to break. I take it that with the 
routes patch things will not fall apart like before?
> 	Read nano.txt from http://www.ssi.bg/~ja/#routes
> It is for 2.4 but the concept and the routing rules are same.
Thanks for that. I already have the setup working as far as multi-homedness 
(that cannot possibly be a word...) is concerned, though.

Thanks for the help.

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Can this be done?

Provided the DS9 patch works for me, is it possible to somehow prioritize what 
is dropped first and what is dropped last? Something like attaching a 
prio/cbq to the ingress qdisc? For example, I want SSH/POP3/IMAP4 traffic to 
be dropped last, and FTP/SMTP traffic to be dropped first. Can ingress 
support in DS9 do this, or is it limited to indiscriminately dropping 
traffic?

Regards.

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

On Thu, 20 Mar 2003, Gordan Bobic wrote:
> > The only problem is that I still didn''t upgraded the patches
to
> > 2.2.25, may be in the next days I''ll find time to do so.
>
> It all patched cleanly anyway, don''t worry about it. I patched
2.2.24 and then
> patched that to 2.2.25. I haven''t compiled it yet because I am not
in front
	OK, I see that skb_padto is used only from the net drivers,
so it is not fatal that skb_copy_grow will not copy skb->tc_index,
so I hope DS9 will work
> Excellent. Note that I also use policy routing so that the sessions started
to
> ethX will always be reponded to on ethX. I think that is fairly standard
> (things break horribly otherwise). However, what happens when a host
randomly
> alternates between IP addresses it is starting sessions to/from? This sort
of
> worked before, but it quickly started to break. I take it that with the
> routes patch things will not fall apart like before?
	Yes, the patch keeps each NAT connection bound to the
masquerade address and correctly uses the routing, usually for
multipath routes. Of course, in 2.2 a route cache flush leads to
much more problems (CPU cycles for rerouting) compared to 2.4.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

On Thu, 20 Mar 2003, Gordan Bobic wrote:
> Provided the DS9 patch works for me, is it possible to somehow prioritize
what
> is dropped first and what is dropped last? Something like attaching a
> prio/cbq to the ingress qdisc? For example, I want SSH/POP3/IMAP4 traffic
to
> be dropped last, and FTP/SMTP traffic to be dropped first. Can ingress
> support in DS9 do this, or is it limited to indiscriminately dropping
> traffic?
	Except if someone has another idea, may be only dropping.
I''m not sure there is IMQ for 2.2 :) May be you can look at
Edge31-ca-u32 from the iproute2 examples, there are some ideas
about using policers. As for DS9, it tries to provide the same
interface to all qos objects as in 2.4, so it is possible to do
the same things as in 2.4, of course, there is one problem, nobody
wants to support new features in 2.2 :)

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Thursday 20 March 2003 23:36, Julian Anastasov wrote:
> Hello,
>
> On Thu, 20 Mar 2003, Gordan Bobic wrote:
> > Provided the DS9 patch works for me, is it possible to somehow
prioritize
> > what is dropped first and what is dropped last? Something like
attaching
> > a prio/cbq to the ingress qdisc? For example, I want SSH/POP3/IMAP4
> > traffic to be dropped last, and FTP/SMTP traffic to be dropped first.
Can
> > ingress support in DS9 do this, or is it limited to indiscriminately
> > dropping traffic?
>
> 	Except if someone has another idea, may be only dropping.
> I''m not sure there is IMQ for 2.2 :)
It doesn''t appear to be there with the patches I applied... :-(
> May be you can look at
> Edge31-ca-u32 from the iproute2 examples, there are some ideas
> about using policers.
I have tried the ingress policing when the new patches, but according to my 
mrtg graphs, it still doesn''t seem to work, even in it''s
simplest form. I
THINK I saw it work with FTP, though (connection throttled), so I am not 
sure. It could be that it works initially, then stops later. I will 
investigate further...
> As for DS9, it tries to provide the same
> interface to all qos objects as in 2.4, so it is possible to do
> the same things as in 2.4, of course, there is one problem, nobody
> wants to support new features in 2.2 :)
Yeah, I know. I will upgrade at some point. ;-)

I seem to be having a problem with the newly applied patches, though. After 
6-8 hours, I get a kernel crash. Black screen od death, hex dumps, Aiee, 
killing interrupt handler, trying to kill idle task, etc. This happened 
twice, so it seems unlikely to be a coincidence, as the only patches applied 
were the DS9, rtree, HTB3 and routes. It used to be stable before. I am 
hoping that it is ingress related stuff that kills it, so I have turned that 
off (everything to do with ingress). We''ll see if I get another crash
now. If
not, than it would appear that it is ingress policing that kills it after a 
while.

I will report back on this when I know more.

Regards.

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

On Sat, 22 Mar 2003, Gordan Bobic wrote:
> I seem to be having a problem with the newly applied patches, though. After
> 6-8 hours, I get a kernel crash. Black screen od death, hex dumps, Aiee,
> killing interrupt handler, trying to kill idle task, etc. This happened
> twice, so it seems unlikely to be a coincidence, as the only patches
applied
> were the DS9, rtree, HTB3 and routes. It used to be stable before. I am
> hoping that it is ingress related stuff that kills it, so I have turned
that
> off (everything to do with ingress). We''ll see if I get another
crash now. If
> not, than it would appear that it is ingress policing that kills it after a
> while.
>
> I will report back on this when I know more.
	ok, it is interesting where is the oops exactly (may be
by avoiding modules and compiling qos in kernel). In any case,
I created patch to upgrade to HTB 3.10
> Regards.
>
> Gordan
Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Sunday 23 March 2003 13:49, Julian Anastasov wrote:
> Hello,
>
> On Sat, 22 Mar 2003, Gordan Bobic wrote:
> > I seem to be having a problem with the newly applied patches, though.
> > After 6-8 hours, I get a kernel crash. Black screen od death, hex
dumps,
> > Aiee, killing interrupt handler, trying to kill idle task, etc. This
> > happened twice, so it seems unlikely to be a coincidence, as the only
> > patches applied were the DS9, rtree, HTB3 and routes. It used to be
> > stable before. I am hoping that it is ingress related stuff that kills
> > it, so I have turned that off (everything to do with ingress).
We''ll see
> > if I get another crash now. If not, than it would appear that it is
> > ingress policing that kills it after a while.
> >
> > I will report back on this when I know more.
>
> 	ok, it is interesting where is the oops exactly (may be
> by avoiding modules and compiling qos in kernel). In any case,
> I created patch to upgrade to HTB 3.10
It would seem that the crash was DEFINITELY caused by ingress filtering. I 
have switched it off (while keeping all egress qdiscs and filters), and now 
the router has been up for 31 hours without a glitch. Oh, well. I will look 
into patching with the latest HTB, but since I don''t use it, there
seems to
be little point...

Unfortunately, I cannot tell in which module the oops happens, because once 
the machine crashes, there are no logs and the start of the error scrolls off 
the screen. :-(

All I can tell for sure is that with ingress used it crashes, without it is 
fine.

Regards.

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hi.

I have just had another very similar crash. It would appear that it is not 
related specifically to ingress policing. It appears to also happen using 
just egress cbq/sfq. However, the uptime between crashes has increased from 
around 6 hours to over 24 hours with ingress filtering switched off, with 
similar (and probably heavier) usage.

Unfortunately, I can still find nothing useful in the logs, and there was 
nothing useful on the screen either (just a hex dump).

I have had this problem (or a very similar one with the same symptoms) before 
with QoS on earlier, unpatched v2.2.x kernels (circa 2.2.14 era). Has anybody 
else experienced/reported anything similar?

I guess I''ll just have to upgrade to v2.4...

Regards.

Gordan
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/