Shorewall users - Mar 2006 - LVS-DR + Shorewall Upgrade 3.0.2 -> 3.0.4 => Trouble

Hello,

after upgrading Shorewall (see subject)  and Gentoo-Linux (from Kernel 2.6.12 
to 2.6.15, both with Gentoo patches, e.g. not Vanilla) the firewall on our 
load balancer rejects  HTTP packets for  the VIP with 
>Mar  5 23:22:51 balance Shorewall:all2all:REJECT:IN= OUT=eth0
>SRC=XX.XXX.XXX.XXX >DST=XXX.XXX.XXX. XXX LEN=48 TOS=0x00 PREC=0x00
TTL=114
>ID=26421 DF PROTO=TCP SPT=2025 DPT=80 >WINDOW=16384 RES=0 
>00 SYN URGP=0
There is only one network card in the balancer (eth0) and DR sends the packets 
via Ethernet, not via NAT/TUN. 
In the  version 3.0.2 it was sufficient to 
a) put the VIP into /etc/shorwall/hosts with
	dmz eth0:XXX.XXX.XXX. XXX
b) put into /etc/shorewall/rules a rule saying
	ACCEPT  net     all     tcp     http

Since it did NOT work to use ''dmz'' with 3.0.2 I assume that we
did something
wrong and it only worked by chance :(.

Nevertheless, I''m baffled. How do I do it the right way in 3.0.4?

Thanks in advance

MvB

PS: Of course the config of LVS inside the Kernel is the same...


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Martin von Boehlen wrote:
> Hello,
> 
> after upgrading Shorewall (see subject)  and Gentoo-Linux (from Kernel
2.6.12
> to 2.6.15, both with Gentoo patches, e.g. not Vanilla) the firewall on our 
> load balancer rejects  HTTP packets for  the VIP with 
>>Mar  5 23:22:51 balance Shorewall:all2all:REJECT:IN= OUT=eth0
>>SRC=XX.XXX.XXX.XXX >DST=XXX.XXX.XXX. XXX LEN=48 TOS=0x00 PREC=0x00
TTL=114
>>ID=26421 DF PROTO=TCP SPT=2025 DPT=80 >WINDOW=16384 RES=0 
>>00 SYN URGP=0
> There is only one network card in the balancer (eth0) and DR sends the
packets
> via Ethernet, not via NAT/TUN. 
> In the  version 3.0.2 it was sufficient to 
> a) put the VIP into /etc/shorwall/hosts with
> 	dmz eth0:XXX.XXX.XXX. XXX
Martin, if you mask IP addresses like that, it''s going to be hard for
us
to help you.  Please at least substitute them with unique ids
consistently so we can correlate.

Also try setting your policies as suggested in
http://sourceforge.net/mailarchive/message.php?msg_id=14807387 or the
latest samples
http://svn.sourceforge.net/viewcvs.cgi/shorewall/trunk/Samples/two-interfaces/policy?view=markup&rev=3620

Paul



-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

On Sunday 05 March 2006 16:12, Martin von Boehlen wrote:
> Hello,
>
> after upgrading Shorewall (see subject)  and Gentoo-Linux (from Kernel
> 2.6.12 to 2.6.15, both with Gentoo patches, e.g. not Vanilla) the firewall
> on our load balancer rejects  HTTP packets for  the VIP with
>
> >Mar  5 23:22:51 balance Shorewall:all2all:REJECT:IN= OUT=eth0
> >SRC=XX.XXX.XXX.XXX >DST=XXX.XXX.XXX. XXX LEN=48 TOS=0x00 PREC=0x00
TTL=114
> >ID=26421 DF PROTO=TCP SPT=2025 DPT=80 >WINDOW=16384 RES=0
> >00 SYN URGP=0
>
> There is only one network card in the balancer (eth0) and DR sends the
> packets via Ethernet, not via NAT/TUN.
> In the  version 3.0.2 it was sufficient to
> a) put the VIP into /etc/shorwall/hosts with
> 	dmz eth0:XXX.XXX.XXX. XXX
> b) put into /etc/shorewall/rules a rule saying
> 	ACCEPT  net     all     tcp     http
>
> Since it did NOT work to use ''dmz'' with 3.0.2 I assume
that we did
> something wrong and it only worked by chance :(.
>
> Nevertheless, I''m baffled. How do I do it the right way in 3.0.4?
>
a) There are no Shorewall changes from 3.0.2->3.0.4 that should cause any 
problems of the sort you describe (there should *never* be any upgrade issues 
when you upgrade from one minor release to another within a major release).

b) You have given us nothing that can help us solve your problem. I dare say 
that no one on the list can describe your setup from what you''ve said
above
-- using ''dmz'' together with ''only one network
card'' in describing a problem
begs explaination.

c) The packets that are being rejected and logged are OUTPUT packets (fw -> 
<some zone>) so the rule that you mention above is completely irrelevant.

Hint -- please see http://www.shorewall.net/support.htm. We are not 

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

Am Monday, 6. March 2006 02:13 schrieb Tom Eastep:
> On Sunday 05 March 2006 16:12, Martin von Boehlen wrote:
> > Hello,
> >
> > after upgrading Shorewall (see subject)  and Gentoo-Linux (from Kernel
> > 2.6.12 to 2.6.15, both with Gentoo patches, e.g. not Vanilla) the
> > firewall on our load balancer rejects  HTTP packets for  the VIP with
> >
> > >Mar  5 23:22:51 balance Shorewall:all2all:REJECT:IN= OUT=eth0
> > >SRC=XX.XXX.XXX.XXX >DST=XXX.XXX.XXX. XXX LEN=48 TOS=0x00
PREC=0x00
> > > TTL=114 ID=26421 DF PROTO=TCP SPT=2025 DPT=80 >WINDOW=16384
RES=0
> > >00 SYN URGP=0
> >
> > There is only one network card in the balancer (eth0) and DR sends the
> > packets via Ethernet, not via NAT/TUN.
> > In the  version 3.0.2 it was sufficient to
> > a) put the VIP into /etc/shorwall/hosts with
> > 	dmz eth0:XXX.XXX.XXX. XXX
> > b) put into /etc/shorewall/rules a rule saying
> > 	ACCEPT  net     all     tcp     http
> >
> > Since it did NOT work to use ''dmz'' with 3.0.2 I
assume that we did
> > something wrong and it only worked by chance :(.
> >
> > Nevertheless, I''m baffled. How do I do it the right way in
3.0.4?
>
> a) There are no Shorewall changes from 3.0.2->3.0.4 that should cause
any
> problems of the sort you describe (there should *never* be any upgrade
> issues when you upgrade from one minor release to another within a major
> release).
>
> b) You have given us nothing that can help us solve your problem. I dare
> say that no one on the list can describe your setup from what
you''ve said
> above -- using ''dmz'' together with ''only one
network card'' in describing a
> problem begs explaination.
>
> c) The packets that are being rejected and logged are OUTPUT packets (fw
->
> <some zone>) so the rule that you mention above is completely
irrelevant.
>
Sorry, for my terse description.  Here is a short explanation of LVS-DR on the 
balancer. It works by using a virtual IP (VIP). I''ve assigned that VIP
to the
single network card (eth0) via /sbin/ip, not via /sbin/ifconfig. Afterwards 
I''m  able to use destinations like $FW:VIP in rules. 

The problem is that to forward packets arriving on said VIP to real servers 
''The load balancer simply changes the MAC address of the data frame to
that
of the chosen server and restransmits it on the LAN. ''  (see 
http://www.linuxvirtualserver.org/VS-DRouting.html). Thus, an incomming 
packet with DST==VIP is changed to an outgoing frame on the (single) Ethernet 
device with a SRC from somewhere on the Internet and a DST==VIP.

With the rule 
ACCEPT  net     all     tcp     http
and the VIP in a zone called ''dmz'' and defined in
/etc/shorewall/hosts
it worked in 3.0.2. Probably not because this is the right way to do it, but 
because of some side-effect.
I''ve set up our backup-balancer to just that configuration and it works
there
still. Removing the rule leads to blocked outgoing frames. Though this is not 
exactly a proof, since this server has more interfaces and more rules than 
the original balancer.

It might be that it works because the re-transmitted frame is regarded as an 
incomming packet in 3.0.2, which seems to be the case no longer with 3.0.4.
But perhaps it has nothing to do with Shorewall at all, but with upgrading the 
Kernel to 2.6.15 ?

MvB
> Hint -- please see http://www.shorewall.net/support.htm. We are not
>
> -Tom


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

On Monday 06 March 2006 03:03, Martin von Boehlen wrote:
>
> The problem is that to forward packets arriving on said VIP to real servers
> ''The load balancer simply changes the MAC address of the data
frame to that
> of the chosen server and restransmits it on the LAN. ''  (see
> http://www.linuxvirtualserver.org/VS-DRouting.html). Thus, an incomming
> packet with DST==VIP is changed to an outgoing frame on the (single)
> Ethernet device with a SRC from somewhere on the Internet and a DST==VIP.
Ah -- that thing. We have never claimed that Shorewall works with it. The 
Shorewall box is only seeing traffic in one direction (inbound) so connection 
tracking doesn''t get a chance to work as it should; Shorewall is based
on
Netfilter''s connection tracking so unpredictable results are certainly 
possible.

At the very least, I would suggest that you add a "fw all ACCEPT"
policy --
You seem unwilling to share your Shorewall configuration with us (and you 
haven''t provided the information asked for at 
http://www.shorewall.net/support.htm) so I''m afraid that is all that I
can
suggest.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

On Monday 06 March 2006 07:15, Tom Eastep wrote:
> On Monday 06 March 2006 03:03, Martin von Boehlen wrote:
> > The problem is that to forward packets arriving on said VIP to real
> > servers ''The load balancer simply changes the MAC address of
the data
> > frame to that of the chosen server and restransmits it on the LAN.
''
> > (see
> > http://www.linuxvirtualserver.org/VS-DRouting.html). Thus, an
incomming
> > packet with DST==VIP is changed to an outgoing frame on the (single)
> > Ethernet device with a SRC from somewhere on the Internet and a
DST==VIP.
>
> Ah -- that thing. We have never claimed that Shorewall works with it. The
> Shorewall box is only seeing traffic in one direction (inbound) so
> connection tracking doesn''t get a chance to work as it should;
Shorewall is
> based on Netfilter''s connection tracking so unpredictable results
are
> certainly possible.
>
> At the very least, I would suggest that you add a "fw all ACCEPT"
policy --
> You seem unwilling to share your Shorewall configuration with us (and you
> haven''t provided the information asked for at
> http://www.shorewall.net/support.htm) so I''m afraid that is all
that I can
> suggest.
I''ve thought about this some more and one possible explaination would
be that
the handling of the altered packets has changed in the kernel. If in the 
earlier kernel, the rewritten packets went through the FORWARD chain rather 
than the OUTPUT chain (which they are doing in the later kernel) then your 
old rule would have worked whereas now it does not. My suggested additional 
policy will allow it to work once again, or you could add a rule "ACCEPT fw
all tcp http".

Just a guess though,
-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

Am Monday, 6. March 2006 17:54 schrieb Tom Eastep:
> On Monday 06 March 2006 07:15, Tom Eastep wrote:
> > On Monday 06 March 2006 03:03, Martin von Boehlen wrote:
> > > The problem is that to forward packets arriving on said VIP to
real
> > > servers ''The load balancer simply changes the MAC
address of the data
> > > frame to that of the chosen server and restransmits it on the
LAN. ''
> > > (see
> > > http://www.linuxvirtualserver.org/VS-DRouting.html). Thus, an
incomming
> > > packet with DST==VIP is changed to an outgoing frame on the
(single)
> > > Ethernet device with a SRC from somewhere on the Internet and a
> > > DST==VIP.
> >
> > Ah -- that thing. We have never claimed that Shorewall works with it.
The
> > Shorewall box is only seeing traffic in one direction (inbound) so
> > connection tracking doesn''t get a chance to work as it
should; Shorewall
> > is based on Netfilter''s connection tracking so unpredictable
results are
> > certainly possible.
> >
> > At the very least, I would suggest that you add a "fw all
ACCEPT" policy
> > -- You seem unwilling to share your Shorewall configuration with us
(and
> > you haven''t provided the information asked for at
> > http://www.shorewall.net/support.htm) so I''m afraid that is
all that I
> > can suggest.
>
> I''ve thought about this some more and one possible explaination
would be
> that the handling of the altered packets has changed in the kernel. If in
> the earlier kernel, the rewritten packets went through the FORWARD chain
> rather than the OUTPUT chain (which they are doing in the later kernel)
> then your old rule would have worked whereas now it does not. My suggested
> additional policy will allow it to work once again, or you could add a rule
> "ACCEPT fw all tcp http".
>
> Just a guess though,
> -Tom
Yes -- that thingy. I know that you''ve never claimed to support it. And
I
agree that it most probably is about changes in Kernel. Furtheremore, it is 
not my idea to X-out IPs and to keep configurations from the list. It is 
company policy. That said I have to state that adding  "ACCEPT $FW all tcp 
http" did change behaviour. It resulted in:

Mar  8 00:11:27 balancer Shorewall:net2all:DROP:IN=eth0 OUT= 
MAC=00:04:75:d2:c6:01:00:04:75:9b:06:5a:08:00 SRC=XXX.XXX.XXX.XX 
DST=XXX.XXX.XXX.XXX LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=495 DF PROTO=ICMP 
TYPE=8 CODE=0 ID=42062 SEQ=6400

This means that instead of the all2all rule the net2all rule is complaining. 
That is a kind of achievement. But not realy useful.

Finally, adding 
ACCEPT  net     all     tcp     http
ACCEPT  all     net     tcp     http

made the grades!

Thanks a lot for putting me on the right track!!

MvB


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Ooops. I did post an ICMP packet here by mistake. But be assured that it was 
the selfsame case with packets for port 80.

Sorry.

MvB

Am Wednesday, 8. March 2006 00:32 schrieb Martin von
Boehlen:
> Am Monday, 6. March 2006 17:54 schrieb Tom Eastep:
> > On Monday 06 March 2006 07:15, Tom Eastep wrote:
> > > On Monday 06 March 2006 03:03, Martin von Boehlen wrote:
> > > > The problem is that to forward packets arriving on said VIP
to real
> > > > servers ''The load balancer simply changes the MAC
address of the data
> > > > frame to that of the chosen server and restransmits it on
the LAN. ''
> > > > (see
> > > > http://www.linuxvirtualserver.org/VS-DRouting.html). Thus,
an
> > > > incomming packet with DST==VIP is changed to an outgoing
frame on the
> > > > (single) Ethernet device with a SRC from somewhere on the
Internet
> > > > and a DST==VIP.
> > >
> > > Ah -- that thing. We have never claimed that Shorewall works with
it.
> > > The Shorewall box is only seeing traffic in one direction
(inbound) so
> > > connection tracking doesn''t get a chance to work as it
should;
> > > Shorewall is based on Netfilter''s connection tracking so
unpredictable
> > > results are certainly possible.
> > >
> > > At the very least, I would suggest that you add a "fw all
ACCEPT"
> > > policy -- You seem unwilling to share your Shorewall
configuration with
> > > us (and you haven''t provided the information asked for
at
> > > http://www.shorewall.net/support.htm) so I''m afraid that
is all that I
> > > can suggest.
> >
> > I''ve thought about this some more and one possible
explaination would be
> > that the handling of the altered packets has changed in the kernel. If
in
> > the earlier kernel, the rewritten packets went through the FORWARD
chain
> > rather than the OUTPUT chain (which they are doing in the later
kernel)
> > then your old rule would have worked whereas now it does not. My
> > suggested additional policy will allow it to work once again, or you
> > could add a rule "ACCEPT fw all tcp http".
> >
> > Just a guess though,
> > -Tom
>
> Yes -- that thingy. I know that you''ve never claimed to support
it. And I
> agree that it most probably is about changes in Kernel. Furtheremore, it is
> not my idea to X-out IPs and to keep configurations from the list. It is
> company policy. That said I have to state that adding  "ACCEPT $FW all
tcp
> http" did change behaviour. It resulted in:
>
> Mar  8 00:11:27 balancer Shorewall:net2all:DROP:IN=eth0 OUT>
MAC=00:04:75:d2:c6:01:00:04:75:9b:06:5a:08:00 SRC=XXX.XXX.XXX.XX
> DST=XXX.XXX.XXX.XXX LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=495 DF PROTO=ICMP
> TYPE=8 CODE=0 ID=42062 SEQ=6400
>
> This means that instead of the all2all rule the net2all rule is
> complaining. That is a kind of achievement. But not realy useful.
>
> Finally, adding
> ACCEPT  net     all     tcp     http
> ACCEPT  all     net     tcp     http
>
> made the grades!
>
> Thanks a lot for putting me on the right track!!
>
> MvB
-- 

Martin von Böhlen
Leiter IT

Qualimedic.com AG
Brückenstr. 1-3
50667 Köln

Tel. +49 (0)221-2705-222
Fax. +49 (0)221-2705-555

DISCLAIMER: This e-mail sent by Qualimedic.com AG may contain 
confidential, legally privileged or trade secret information 
which is protected by law. It is intended for the named recipient only.
If you are not the intended recipient, please notify us by 
e-mail, confirming that it has been deleted from your system and any hard 
copies destroyed.
You must not use, disclose or distribute, copy, print or rely on 
any information contained in this e-mail.

We do not, to the extent permitted by law, accept any liability 
(whether in contract, negligence or otherwise) for any virus 
infection and/or external compromise of security and/or 
confidentiality in relation to transmissions sent by e-mail.

As with any internet communication, this one may not be secure 
against interception or modification.



-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642