Shorewall users - Oct 2004 - After shorewall restart NAT SMTP connection slow; reboot and it works fine

I recently implemented v2.0.9 using ''shorewall setup guide''
2004-07-31.

Starting with block everything not known to be in use and opening ports
as complaints come in.  This has led to a few rule changes.  After a
rule change I use shorewall restart to reload the rules.  Seems to work
OK... except for an outbound NAT SMTP connection from a mail server on
.122 to postini.com.  The SMTP server can connect and transfer messages
but does so very slowly.  If I reboot the firewall (after the changed
rules are saved) SMTP flows normally.  (Stopping and restarting, or even
rebooting the SMTP server doesn''t help.)  After perusing the
Documentation Index I was wondering if instead of ''shorewall
restart''
the command to use would be ''shorewall refresh'', or maybe
''shorewall
stop; shorewall clear; shorewall start''.  It seems as if there is maybe
a state or translation table somewhere that isn''t being cleared (or
something like that) but that is cleared by a reboot.

Any help sincerely appreciated.  I would prefer not to reboot the
firewall every time there is a rule change, although once we get through
another week there shouldn''t be the same volume of rule changes.

Using shorewall 2.0.9 on latest Bering (LEAF), downloaded updated
shorewall.lrp from link on shorewall site
ip addr show:
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:50:8b:e0:7d:15 brd ff:ff:ff:ff:ff:ff
    inet 12.4.123.114/28 brd 12.4.123.127 scope global eth0
    inet 12.4.123.118/28 brd 12.4.123.127 scope global secondary eth0:0
    inet 12.4.123.122/28 brd 12.4.123.127 scope global secondary eth0:0
4: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:50:8b:e0:7d:14 brd ff:ff:ff:ff:ff:ff
    inet 172.16.2.254/16 brd 172.16.255.255 scope global eth1
ip route:
12.4.123.112/28 dev eth0  proto kernel  scope link  src 12.4.123.114
172.16.0.0/16 dev eth1  proto kernel  scope link  src 172.16.2.254
172.16.0.0/13 via 172.16.1.1 dev eth1
default via 12.4.123.113 dev eth0

Tom - thanks for Shorewall.


Glenn St. John
THIS MESSAGE contains information that is CONFIDENTIAL and legally exempt from
disclosure and that is intended only for the addressee of this message.  IF YOU
are not the addressee or a person responsible for delivering this message to the
addressee, then do not copy or deliver this message to anyone and please delete
this message and contact us at +1.703.836.6620.  Thank you.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Glenn St. John wrote:
> I recently implemented v2.0.9 using ''shorewall setup
guide'' 2004-07-31.
>
> Starting with block everything not known to be in use and opening ports
> as complaints come in.  This has led to a few rule changes.  After a
> rule change I use shorewall restart to reload the rules.  Seems to work
> OK... except for an outbound NAT SMTP connection from a mail server on
> .122 to postini.com.  The SMTP server can connect and transfer messages
> but does so very slowly.  If I reboot the firewall (after the changed
> rules are saved) SMTP flows normally.  (Stopping and restarting, or even
> rebooting the SMTP server doesn''t help.)  After perusing the
> Documentation Index I was wondering if instead of ''shorewall
restart''
> the command to use would be ''shorewall refresh'', or maybe
''shorewall
> stop; shorewall clear; shorewall start''.  It seems as if there is
maybe
> a state or translation table somewhere that isn''t being cleared
(or
> something like that) but that is cleared by a reboot.
>
> Any help sincerely appreciated.  I would prefer not to reboot the
> firewall every time there is a rule change, although once we get through
> another week there shouldn''t be the same volume of rule changes.
>
The "shorewall restart" command completely reloads the Netfilter
configuration and all of us who use Shorewall will routinely alter our
ruleset and use "shorewall restart" to activate the change. To my
knowledge, no one has seen ever reported the behavior that you are seeing.

If you can reproduce this behavior again, I suggest that you:

a) Note the conntrack table entry/entries for the slow connection(s)
(from "shorewall show connections").
b) look at the slow connection with a packet sniffer such as Ethereal or
tcpdump. That may give you a hint.
c) After the reboot, and while mail is flowing normally look at the
conntrack table entry/entries again.

I mention the connection tracking table because that table is *not*
cleared by a "shorewall restart" but an incorrect conntrack table
entry
usually causes things to not work at all and would not produce the
symptoms that you report.

And before you ask "why doesn''t shorewall clear the table during
restart", there are 100s of posts covering this topic on the netfilter
user''s list; basically, there is no reliable way to do it other than to
reboot.

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBd9kVO/MAbZfjDLIRArgUAJ461QwcRNnVwypD88zAP0kxPyzNTACgkCtV
3D6ZF+AgUoiUE2szmLjFOZ0=I5MN
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Eastep wrote:
>
>
> The "shorewall restart" command completely reloads the Netfilter
> configuration and all of us who use Shorewall will routinely alter our
> ruleset and use "shorewall restart" to activate the change. To my
> knowledge, no one has seen ever reported the behavior that you are seeing.
>
> If you can reproduce this behavior again, I suggest that you:
>
> a) Note the conntrack table entry/entries for the slow connection(s)
> (from "shorewall show connections").
> b) look at the slow connection with a packet sniffer such as Ethereal or
> tcpdump. That may give you a hint.
> c) After the reboot, and while mail is flowing normally look at the
> conntrack table entry/entries again.
>
> I mention the connection tracking table because that table is *not*
> cleared by a "shorewall restart" but an incorrect conntrack table
entry
> usually causes things to not work at all and would not produce the
> symptoms that you report.
>
> And before you ask "why doesn''t shorewall clear the table
during
> restart", there are 100s of posts covering this topic on the netfilter
> user''s list; basically, there is no reliable way to do it other
than to
> reboot.
Another possibility is that "shorewall restart" is altering your
configuration in some way that you don''t expect. Do you make use of
ADD_SNAT_ALIASES or ADD_IP_ALIASES in shorewall.conf?

- -Tom
_______________________________________________
Shorewall-users mailing list
Post: Shorewall-users@lists.shorewall.net
Subscribe/Unsubscribe:
https://lists.shorewall.net/mailman/listinfo/shorewall-users
Support: http://www.shorewall.net/support.htm
FAQ: http://www.shorewall.net/FAQ.htm

- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBd+oOO/MAbZfjDLIRAtziAKCig0Px4RbseR4M4attc/a2eCmczACgwwAO
q2k7YxI6MExcFJzn2dx6j58=qNdV
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Eastep wrote:
>
> Another possibility is that "shorewall restart" is altering your
> configuration in some way that you don''t expect. Do you make use
of
> ADD_SNAT_ALIASES or ADD_IP_ALIASES in shorewall.conf?
>
You can see what "shorewall [re]start" does in addition to setting up
Netfilter by looking at /var/lib/shorewall/restore-base. That script is
created during "shorewall [re]start" and is combined with the output
of
"iptables-save" by the "shorewall save" command.

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBd/D3O/MAbZfjDLIRAipTAJ9VUKuegjk7a2woxx/u2hye7zOBQQCeNc3+
EccbfyfdlBzIvCk0Rpga8+M=ME73
-----END PGP SIGNATURE-----

> If you can reproduce this behavior again, I suggest that you:
>
> a) Note the conntrack table entry/entries for the slow connection(s)
> (from "shorewall show connections").
It seems reproducible (three times it has happened) so I will check the
conntrack table and use ethereal to look at some packets.
>Another possibility is that "shorewall restart" is altering your
>configuration in some way that you don''t expect. Do you make use of
>ADD_SNAT_ALIASES or ADD_IP_ALIASES in shorewall.conf?
ADD_IP_ALIASES=Yes in shorewall.conf, (ADD_SNAT_ALIASES=No)
In some way restarting is breaking this or the alias doesn''t work
correctly after the restart?  (I don''t follow where you are headed.)  I
guess the hint you gave me was to look at
/var/lib/shorewall/restore-base, I see under ''restoring one-to-one
NAT..'' it deletes the addresses? and then under ''restoring ip
addresses...'' it adds them back ?  In all likelihood I''m
mis-reading the
script so any help there appreciated.
> And before you ask "why doesn''t shorewall clear the table
during
> restart", there are 100s of posts covering this topic on the netfilter
Simple things should be simple and complex things possible (I read that
somewhere) but I don''t expect shorewall to do something impossible.  If
it turns out in my environment, on leaf, with NAT, etc - that the answer
is to reboot to clear the conntrack table to implement a rule change
then so be it.  It takes about 2 minutes to reboot which is longer than
reloading but still not too bad.

Glenn St. John

THIS MESSAGE contains information that is CONFIDENTIAL and legally exempt from
disclosure and that is intended only for the addressee of this message.  IF YOU
are not the addressee or a person responsible for delivering this message to the
addressee, then do not copy or deliver this message to anyone and please delete
this message and contact us at +1.703.836.6620.  Thank you.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Glenn St. John wrote:
>
>
> ADD_IP_ALIASES=Yes in shorewall.conf, (ADD_SNAT_ALIASES=No)
> In some way restarting is breaking this or the alias doesn''t work
> correctly after the restart?  (I don''t follow where you are
headed.)  I
> guess the hint you gave me was to look at
> /var/lib/shorewall/restore-base, I see under ''restoring one-to-one
> NAT..'' it deletes the addresses? and then under
''restoring ip
> addresses...'' it adds them back ?  In all likelihood I''m
mis-reading the
> script so any help there appreciated.
That''s what is happening -- so the real key is "which address(es)
are
being deleted?". If these are addresses with routes associated with them
then when Shorewall adds back the address, it has no way of adding back
the route. So in particular, if you try to use one-to-one NAT on the
primary IP address of an external interface, these sorts of problems occur.
>
>
>>And before you ask "why doesn''t shorewall clear the table
during
>>restart", there are 100s of posts covering this topic on the
netfilter
>
>
> Simple things should be simple and complex things possible (I read that
> somewhere) but I don''t expect shorewall to do something
impossible.  If
> it turns out in my environment, on leaf, with NAT, etc - that the answer
> is to reboot to clear the conntrack table to implement a rule change
> then so be it.  It takes about 2 minutes to reboot which is longer than
> reloading but still not too bad.
As I said before, no one else that I''m aware of is rebooting to install
a new ruleset -- I certainly wouldn''t put up with doing that.

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBeBWeO/MAbZfjDLIRArizAKCrwJwNbRUSfDPOpObiq5iOQSwsygCgjmaf
sETORKeeWNFtxOxRktZuXi8=0H8g
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Eastep wrote:
> Glenn St. John wrote:
>
>
>>>
>>>ADD_IP_ALIASES=Yes in shorewall.conf, (ADD_SNAT_ALIASES=No)
>>>In some way restarting is breaking this or the alias
doesn''t work
>>>correctly after the restart?  (I don''t follow where you are
headed.)  I
>>>guess the hint you gave me was to look at
>>>/var/lib/shorewall/restore-base, I see under ''restoring
one-to-one
>>>NAT..'' it deletes the addresses? and then under
''restoring ip
>>>addresses...'' it adds them back ?  In all likelihood
I''m mis-reading the
>>>script so any help there appreciated.
>
>
> That''s what is happening -- so the real key is "which
address(es) are
> being deleted?". If these are addresses with routes associated with
them
> then when Shorewall adds back the address, it has no way of adding back
> the route. So in particular, if you try to use one-to-one NAT on the
> primary IP address of an external interface, these sorts of problems
occur.
>
>
>>>
>>>>And before you ask "why doesn''t shorewall clear
the table during
>>>>restart", there are 100s of posts covering this topic on
the netfilter
>>>
>>>
>>>Simple things should be simple and complex things possible (I read
that
>>>somewhere) but I don''t expect shorewall to do something
impossible.  If
>>>it turns out in my environment, on leaf, with NAT, etc - that the
answer
>>>is to reboot to clear the conntrack table to implement a rule change
>>>then so be it.  It takes about 2 minutes to reboot which is longer
than
>>>reloading but still not too bad.
So I would compare:

a) The output of "ip addr ls" before and after you restart Shorewall.
b) The output of "ip route ls" before and after you restart Shorewall.

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBeBYCO/MAbZfjDLIRAl9NAKC6Jg4V5jBMSyH3GxUmrgy5qZuC7gCdE/TE
L5rpOGwaDZ7hdGbf0DSAtfA=/2w9
-----END PGP SIGNATURE-----