thr3ads.net - Shorewall users - Dual-homing BGP gate problem [Dec 2009]

If this information is useful, please help other people find it:
Share via:

Andrzej Odyniec

2009-Dec-16 11:09 UTC

Dual-homing BGP gate problem

Hi Tom,

After two weeks of nightmares I decided ask You (and anyone reading this mail).

Context is as follows:

I try to update system on my central router from kernel 2.6.29.6 and Shorewall 
4.2.6 (old) to kernel 2.6.31.6 and Shorewall 4.4.4.2 (new).

This is LiveCD image boot (Devil-Linux distribution compiled by me), so config 
is this same.

I have established ten OpenVPN tunnels and two IPsec tunnels. I have five 
ethernet interfaces, to LAN, DMZ, WLAN and two to ISPs.

With ISPs BGP sessions (quagga) are established. BGP sessions are simple 
dual-homing, broadcasting only my "C" class, with deny on input only
rfc1918
addresses, without transit and any other message manipulations.

Providers are declared in /etc/shorewall/providers but without marks (only for 
generate provider routing tables). There are no any tc declarations.

After change of OS some tunnels cannot be reestablished because FROM some 
divisions clients cannot connect to this central router.

Two weeks of night testing gives interesting results! Problems with connection 
to VPN servers on my router are only from this sites, from where routefilter 
should discard answer packets (packet enters via one ISP and exits via 
another, call this assymetric). But my ISP interfaces are declared with 
options:  optional,logmartians,blacklist,nosmurfs,routefilter=0
and in /proc/sys/net this really is set. Problem is not with answers to router 
(as we can think), but with answer to external connections, so I can ping from 
router this external localization.

If packet enters via this same interface as exits (symmetric), all is working 
in new system as was in old one.

In /proc/sys/net settings are this same except 
/proc/sys/net/ipv4/conf/.../arp_accept=0 not present in old kernel.

Additionally I tested on old system new quagga version (0.99.15 as in new 
system) -- was working OK.

On new system I run iptrace. For symmetric ICMP packet from external source to 
my gate 195.187.140.1 I obtain:
> TRACE: raw:PREROUTING:policy:2 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:PREROUTING:rule:1 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:tcpre:return:1 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:PREROUTING:policy:2 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:INPUT:policy:1 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: filter:INPUT:rule:1 IN=eth3 OUT= SRC=83.3.197.202 DST=195.187.140.1
LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8 CODE=0 ID=49174
SEQ=37889
> TRACE: filter:dynamic:return:1 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: filter:INPUT:rule:3 IN=eth3 OUT= SRC=83.3.197.202 DST=195.187.140.1
LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8 CODE=0 ID=49174
SEQ=37889
> TRACE: filter:eth3_in:rule:1 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: filter:blacklst:return:4 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: filter:eth3_in:rule:3 IN=eth3 OUT= SRC=83.3.197.202
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8
CODE=0 ID=49174 SEQ=37889
> TRACE: filter:net2fw:rule:1 IN=eth3 OUT= SRC=83.3.197.202 DST=195.187.140.1
LEN=28 TOS=0x00 PREC=0x00 TTL=59 ID=48172 PROTO=ICMP TYPE=8 CODE=0 ID=49174
SEQ=37889
> ====> TRACE: raw:OUTPUT:policy:2 IN= OUT=eth3 SRC=195.187.140.1
DST=83.3.197.202 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:OUTPUT:rule:1 IN= OUT=eth3 SRC=195.187.140.1 DST=83.3.197.202
LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0 CODE=0 ID=49174
SEQ=37889
> TRACE: mangle:tcout:return:1 IN= OUT=eth3 SRC=195.187.140.1
DST=83.3.197.202 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:OUTPUT:policy:2 IN= OUT=eth3 SRC=195.187.140.1
DST=83.3.197.202 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0
CODE=0 ID=49174 SEQ=37889
> TRACE: filter:OUTPUT:rule:2 IN= OUT=eth3 SRC=195.187.140.1 DST=83.3.197.202
LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0 CODE=0 ID=49174
SEQ=37889
> TRACE: filter:eth3_out:rule:1 IN= OUT=eth3 SRC=195.187.140.1
DST=83.3.197.202 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0
CODE=0 ID=49174 SEQ=37889
> TRACE: filter:fw2net:rule:1 IN= OUT=eth3 SRC=195.187.140.1 DST=83.3.197.202
LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0 CODE=0 ID=49174
SEQ=37889
> TRACE: mangle:POSTROUTING:rule:1 IN= OUT=eth3 SRC=195.187.140.1
DST=83.3.197.202 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:tcpost:return:1 IN= OUT=eth3 SRC=195.187.140.1
DST=83.3.197.202 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0
CODE=0 ID=49174 SEQ=37889
> TRACE: mangle:POSTROUTING:policy:2 IN= OUT=eth3 SRC=195.187.140.1
DST=83.3.197.202 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=33445 PROTO=ICMP TYPE=0
CODE=0 ID=49174 SEQ=37889
and router respond correctly to ping.

But for asymmetric one (for this localization output route is via eth3 too but 
packet arrives via eth2) after PREROUTING chain in mangle table ping packet 
enters PREROUTING chain in nat table (there is no nat rule for this addresses) 
and get stuck:
> TRACE: raw:PREROUTING:policy:2 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
> TRACE: mangle:PREROUTING:rule:1 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
> TRACE: mangle:tcpre:return:1 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
> TRACE: mangle:PREROUTING:policy:2 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
> TRACE: nat:PREROUTING:rule:1 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
> TRACE: nat:dnat:rule:1 IN=eth2 OUT= SRC=89.174.215.22 DST=195.187.140.1
LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8 CODE=0 ID=12662
SEQ=260
> TRACE: nat:net_dnat:return:30 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
> TRACE: nat:dnat:return:26 IN=eth2 OUT= SRC=89.174.215.22 DST=195.187.140.1
LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8 CODE=0 ID=12662
SEQ=260
> TRACE: nat:PREROUTING:policy:2 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
I suspect changed behavior of new kernel or new Shorewall.

I cannot test new Shorewall with old system.

There is side effect in new system: two daemons looking for ARP (arpwatch and 
ntop) in new system are consuming all processor cores time.

Maybe You know any settings solving this.

Best Regards

-- 
Andrzej Odyniec

<anody@macrologic.pl>
Rada Nadzorcza Macrologic SA
ul. Chrościckiego 49, 02-414 Warszawa
tel. +48(22)8637681x132, kom. +48(601)276572
ul. Kłopotowskiego 22, 03-717 Warszawa
tel. +48(22)5118115, fax: +48(22)5118117
Rejestr: Sąd Rejonowy dla m.st. Warszawy,
          XIII Wydział Gospodarczy Krajowego
          Rejestru Sądowego, numer 0000045462
Numer identyfikacji podatkowej: PL 522-000-28-25
Kapitał zakładowy: 1888719 zł opłacony w całości


------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon''s best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

Tom Eastep

2009-Dec-16 15:08 UTC

head link

Re: Dual-homing BGP gate problem

On 12/16/2009 03:09 AM, Andrzej Odyniec wrote:> 
> But for asymmetric one (for this localization output route is via eth3 too
but
> packet arrives via eth2) after PREROUTING chain in mangle table ping packet
> enters PREROUTING chain in nat table (there is no nat rule for this
addresses)
> and get stuck:
> 
>> TRACE: raw:PREROUTING:policy:2 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
>> TRACE: mangle:PREROUTING:rule:1 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
>> TRACE: mangle:tcpre:return:1 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
>> TRACE: mangle:PREROUTING:policy:2 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
>> TRACE: nat:PREROUTING:rule:1 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
>> TRACE: nat:dnat:rule:1 IN=eth2 OUT= SRC=89.174.215.22 DST=195.187.140.1
LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8 CODE=0 ID=12662
SEQ=260
>> TRACE: nat:net_dnat:return:30 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
>> TRACE: nat:dnat:return:26 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
>> TRACE: nat:PREROUTING:policy:2 IN=eth2 OUT= SRC=89.174.215.22
DST=195.187.140.1 LEN=28 TOS=0x00 PREC=0x00 TTL=55 ID=49071 PROTO=ICMP TYPE=8
CODE=0 ID=12662 SEQ=260
> 
> I suspect changed behavior of new kernel or new Shorewall.
There is no way that I know of that Shorewall could make this happen.
Have you compared the output of ''shorewall show nat'' on the
two
different versions?

-Tom
-- 
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon''s best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

Andrzej Odyniec

2009-Dec-16 16:50 UTC

head link

Re: Dual-homing BGP gate problem

Dear Tom,

Thanks for answer.

Tom Eastep wrote:>>I suspect changed behavior of new kernel or new Shorewall.
> 
> There is no way that I know of that Shorewall could make this happen.
> Have you compared the output of ''shorewall show nat'' on
the two
> different versions?
Good idea. I should compare this.

I made shorewall dump tonight (new) and now (old). NAT tables are identical 
with accuracy to counters.

I compared now side by side all compiled tables in dumps.

I don''t see important differences. In filter INPUT 4.4.4.2 eth2_in and
eth3_in
are called earlier and at end rest is Reject''ed instead of
Drop''ped and
reject''ed instead of DROP''ped (as in 4.2.6).

In FORWARD and OUTPUT ISP interfaces chains are called earlier. Tunnel 
acceptances in net2fw are shifted too. But rest is identical.

Interfaces configurations, especially ISPs, are the same.

So problem is rather not in Shorewall but in netfilter, netfilter patches or 
kernel with patches (iproute2?). Maybe grsec?

I will check it.

Thanks for help

Regards

Andrzej Odyniec

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon''s best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

Andrzej Odyniec

2009-Dec-20 01:40 UTC

head link

Re: Dual-homing BGP gate problem

Dear Tom,

I, Andrzej Odyniec wrote:> Tom Eastep wrote:
>>>I suspect changed behavior of new kernel or new Shorewall.
>>
>>There is no way that I know of that Shorewall could make this happen.
>>Have you compared the output of ''shorewall show nat''
on the two
>>different versions?
[...]> So problem is rather not in Shorewall but in netfilter, netfilter patches
or
> kernel with patches (iproute2?). Maybe grsec?
New kernel: 2.6.31.7-grsec solved problem, but not automatically... just after 
change /proc/sys/net/ipv4/conf/all/rp_filter to "0". rp_filter on
interfaces I
have "0".

In /etc/shorewall/shorewall.conf I have from always ROUTE_FILTER=No.

But in compiled /var/lib/shorewall/filter is in setup_common_rules:

     echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter

It seems, changes in ip routing kernel module changed behavior of 
/proc/sys/net/ settings deeper, than we expect. So there is not only new value 
"2" of rp_filter, but "all" settings is enabling rp_filter
on interfaces. I
think, for understanding we need read Kuznietsov code in kernel. And I think, 
it is possibility to exotic dance of this behavior in new kernels.

So I temporarily added to /etc/shorewall/start
     echo 0 >/proc/sys/net/ipv4/conf/all/rp_filter
and now in 2.6.31 kernel with patch set 7 all is working OK.

In 2.6.29 kernels this was not necessary.

Regards

Andrzej Odyniec

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon''s best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev

Apparently Analagous Threads

Search for more possibly parallel threads

Shorewall users - Dec 2009 - Dual-homing BGP gate problem

Dual-homing BGP gate problem

Re: Dual-homing BGP gate problem

Re: Dual-homing BGP gate problem

Re: Dual-homing BGP gate problem

Apparently Analagous Threads