I''m havign a HUGE amount of difficulty getting shoreline to work with LVS. We use it here constantly so we know it works. The problem is packets come in, get directed to a webserver, webserver returns the packet to firewall, and then it goes into a black hole. rp_filter is off globally on all interfaces. LVS seems to be working right.... I use shorewall tcrules to mark packets on their way in (in the PREROUTING chain) just like I do with our current load balancer. I''m just really confused as to where the packets are being dropped int he firewall... Anyone ever used a setup like this, any ideas? -- GPG/PGP --> 0xE736BD7E 5144 6A2D 977A 6651 DFBE 1462 E351 88B9 E736 BD7E -- Undocumented Features quote of the moment... "It''s not the one bullet with your name on it that you have to worry about; it''s the twenty thousand-odd rounds labeled `occupant.''" --Murphy''s Laws of Combat
On Thu, 2004-11-04 at 21:35, Michael Loftis wrote:> I''m havign a HUGE amount of difficulty getting shoreline to work with LVS. > We use it here constantly so we know it works. The problem is packets come > in, get directed to a webserver, webserver returns the packet to firewall, > and then it goes into a black hole. rp_filter is off globally on all > interfaces. LVS seems to be working right.... > > I use shorewall tcrules to mark packets on their way in (in the PREROUTING > chain) just like I do with our current load balancer. I''m just really > confused as to where the packets are being dropped int he firewall... > > Anyone ever used a setup like this, any ideas?It is my understanding that standard Netfilter and LVS are incompatible -- see http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.filter_rules.html. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
On Fri, 2004-11-05 at 06:57, Tom Eastep wrote:> On Thu, 2004-11-04 at 21:35, Michael Loftis wrote: > > I''m havign a HUGE amount of difficulty getting shoreline to work with LVS. > > > > Anyone ever used a setup like this, any ideas? > > It is my understanding that standard Netfilter and LVS are incompatible > -- see > http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.filter_rules.html.Or at least standard Netfilter connection tracking and LVS appear incompatible which is why I believe that your packets are being dropped -- there seems to be a kernel patch to make Netfilter and LVS play together but I have no clue what sort of Shorewall configuration might work with this combination given the bizarre path that LVS-redirected packets apparently take through Netfilter. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
--On Friday, November 05, 2004 07:19 -0800 Tom Eastep <teastep@shorewall.net> wrote:> Or at least standard Netfilter connection tracking and LVS appear > incompatible which is why I believe that your packets are being dropped > -- there seems to be a kernel patch to make Netfilter and LVS play > together but I have no clue what sort of Shorewall configuration might > work with this combination given the bizarre path that LVS-redirected > packets apparently take through Netfilter.Alright, and here I was beginning to wonder if I did something stupid :) And indeed that seems to be the case here, rules should be allowing in and out traffic but due to ocnnection tracking and vs not talking they''re pretty clearly not. I''ll see if I can find that patch. BTW Tom, thanks once again for the excellent software. Maintaining my firewall would be a nightmare without it :) (six zones, 12+ interfaces, and around 10-15 subnets, and growing!)
--On Friday, November 05, 2004 07:19 -0800 Tom Eastep <teastep@shorewall.net> wrote:> Or at least standard Netfilter connection tracking and LVS appear > incompatible which is why I believe that your packets are being dropped > -- there seems to be a kernel patch to make Netfilter and LVS play > together but I have no clue what sort of Shorewall configuration might > work with this combination given the bizarre path that LVS-redirected > packets apparently take through Netfilter.The whole netfilter path is a mystery to me, I''ve yet to see a single document on it. The little bit I know comes from following code in the kernel, and that isn''t much. I''m still having no luck even after applying the patch, no small feat since I have to do a border firewall upgrade to even test it.... I''m going to try the patch author too, but if anyone on the list has any ideas, i''m all ears. Exact same symptoms, Syn''s make it to the servers, but the ACKs from the servers just go to the garbage, so the three way never completes. I even tried switching from ''MARK'' type services to ip:port services to test, no luck. I think it''s stupid and ridiculous that netfilter and LVS don''t work together.
Michael Loftis wrote:> > The whole netfilter path is a mystery to me, I''ve yet to see a single > document on it. The little bit I know comes from following code in the > kernel, and that isn''t much.http://shorewall.net/NetfilterOverview.html tries to give enough information about the Netfilter flow to understand how Shorewall works. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Hi everyone, as this is a rather old thread I will simply say that I got Netfilter/Shorewall and LVS (NAT / 2.6.9 Kernel) working for my rather basic needs. As this is off topic I will just keep this short. If someone should be interested how it worked and what patch I used just reply to this thread. Regards, -- Axel Westerhold Congos Inc. Technical Lead Tel: (+49) 5732 688040 Cell: (+49) 171 9754 756 PK: 1EF597FA Michael Loftis wrote:> > > --On Friday, November 05, 2004 07:19 -0800 Tom Eastep > <teastep@shorewall.net> wrote: > >> Or at least standard Netfilter connection tracking and LVS appear >> incompatible which is why I believe that your packets are being dropped >> -- there seems to be a kernel patch to make Netfilter and LVS play >> together but I have no clue what sort of Shorewall configuration might >> work with this combination given the bizarre path that LVS-redirected >> packets apparently take through Netfilter. > > > The whole netfilter path is a mystery to me, I''ve yet to see a single > document on it. The little bit I know comes from following code in > the kernel, and that isn''t much. > > I''m still having no luck even after applying the patch, no small feat > since I have to do a border firewall upgrade to even test it.... I''m > going to try the patch author too, but if anyone on the list has any > ideas, i''m all ears. Exact same symptoms, Syn''s make it to the > servers, but the ACKs from the servers just go to the garbage, so the > three way never completes. > > I even tried switching from ''MARK'' type services to ip:port services > to test, no luck. I think it''s stupid and ridiculous that netfilter > and LVS don''t work together. > _______________________________________________ > Shorewall-users mailing list > Post: Shorewall-users@lists.shorewall.net > Subscribe/Unsubscribe: > https://lists.shorewall.net/mailman/listinfo/shorewall-users > Support: http://www.shorewall.net/support.htm > FAQ: http://www.shorewall.net/FAQ.htm
On Fri, 2004-12-03 at 17:21 +0100, Axel Westerhold wrote:> as this is a rather old thread I will simply say that I got > Netfilter/Shorewall and LVS (NAT / 2.6.9 Kernel) working for my rather > basic needs. As this is off topic I will just keep this short. If > someone should be interested how it worked and what patch I used just > reply to this thread.Alex, I think it would be good to have this information recorded here in the archives as I''m sure that it will come up again. Thanks, -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Ok,
ask Tom thinks this might be something to archiv to avoid some further
questions here is what I did to get Netfilter/Shorewall working on my
two LVS Servers.
First of all, this is what I had to get done:
2 to x Mail Relay Servers with identical software setup hosting 5 to x
customers in a setup giving those customers 99% availibility. The
hardware of the servers might vary from highend HP Proliants down to
older Compaq Proliants. We decided to use LVS (NAT) to get enough
scaling capability and a possibility to cluster some reverse proxies.
For some reasons we also placed those systemes within an IP range which
is not covered by our firewall cluster.
|
| Internet Backbone
|
---------
| | Official IP''s
------ ------
| B | | B |
| 1 | | 2 |
------ ------
| |
---------
|
---------
| | RFC 1918 IP''s
------ ------
| R | | R |
| 1 | | 2 |
------ ------
B=Balancer R=Real
The Balancers looks like this
Base: Fedora Core 2 (modified so that the ISO will
install without any questions and within 10 min.)
(Actually this all is mostly depending on the
kernel used the distribution should not matter much).
Kernel: Modified 2.6.9 based on Fedora 2.6.9-1.681_FC2.src.rpm
(This should work on any 2.6.9 kernel and there
are patches for other 2.6.8, 2.6.5, 2.4.27,2.4.26,2.4.23 too.
I tested this using 2.6.9 so no idea how the
others are behaving)
Patch: ipvs-nfct-2.6.9-2.diff
<http://www.ssi.bg/%7Eja/nfct/ipvs-nfct-2.6.9-2.diff>
and ip_nat_ftp-2.6.9-1.diff
<http://www.ssi.bg/%7Eja/nfct/ip_nat_ftp-2.6.9-1.diff>
iptables: v1.2.9 (from fedora)
shorewall: 2.0.11
ipvsadm: 1.2.0 (from fedora)
heartbeat: 1.2.3-1 (compiled from src and rpmbuild)
ldirector: 1.2.3-1 (compiled from src and rpmbuild)
heartbeat and ldirectord can be found here:
http://www.linux-ha.org
The patches can be found here:
http://www.ssi.bg/~ja/nfct
Some additional info can be found here:
http://www.ultramonkey.org/
I used the src.rpm to get the patched kernel install as easily as
possible on machines which have no development tools installs and would
be too slow at all to compile on them. I actually installed the src.rpm,
moved the two patches into the SOURCE folder, added the needed Patchx
aund %patchx entries and did my rpmbuild -bb -target= .
I installed those kernels on my two balancers, installed the shorewall
rpm,heartbeat and ldirectord. Next I modified the needed config files
and added following commands into
/etc/shorewall/initdone
/sbin/ipvsadm
echo 1 > /proc/sys/net/ipv4/vs/conntrack
echo 1 > /proc/sys/net/ipv4/vs/snat_reroute
initdone might not be the best place to do it but as it worked I spared
my time looking at the shorewall docs for the proper place.
The first command is needed to create the proc entry ''vs'' as
this is
only available after ipvsadm run once . Next you will enable the sync
between the LVS contrack and the Netfilter contrack. The last command is
just useful for SNAT and is not directly related to the contract
problem. As I have to do SNAT so each server (having a RFC1918 IP) can
be accessed by a real IP (without balancing) for maintance.
After done and reboot everything worked incl. shorewall DNAT, SNAT and
contracks. (Well, actually it didn''t because I used two virtual
IP''s to
do maintance on the real servers NATing them and forgot to set
ADD_IP_ALIASES in shorewall.conf to NO. That way I suddenly had both
IP''s active on both balancers with the expected result)
Some comments:
Doing it without the patch and shorewall:
I got this all working without the patch and shorewall by simply binding
all managment ports (like ssh) on the internal interface of each
balancer and just keep the SMTP Port on the outside (the virtual IP''s
actually). Then I used simple DNAT and SNAT iptable entries with s=
defining my managment station IP''s to access the balancers and the real
servers. That way nmap and nessus weren''t able to see any other open
ports then SMTP. This setup was working because contrack modules
weren''t needed, but keeping this running with 99% availibility and not
everyone doing maintance was following up on the install process we
decided that doing security not by obscurity and keep with shorewall.
State Syncronisation:
LVS provides a way to sync connection states between clustered
balancers. Keep in mind that this works only for LVS. Even the patch
will not sync the Netfilter contrack in case of a resource takeover.
Stabability:
I had a look at the patches and for me they looked fine. Still, this is
an unofficial patch so be aware that it might or not have problems. The
system currently handles 15k connections a day maybe 20k by now. So I
can''t say if it is really stable under stress.
Axel
Axel Westerhold wrote:
> Hi everyone,
>
> as this is a rather old thread I will simply say that I got
> Netfilter/Shorewall and LVS (NAT / 2.6.9 Kernel) working for my rather
> basic needs. As this is off topic I will just keep this short. If
> someone should be interested how it worked and what patch I used just
> reply to this thread.
>
> Regards,
>