Shorewall users - Sep 2004 - masq - pings and connections get dropped after PREROUTING?

Hello,

I have a pretty standard two-interface setup with masquerading, so the local
network can connect through the firewall to the Internet.

On the firewall box (trevor), eth0 is connected to a cable modem and eth1 is
connected to the local network via a crossed cable.  There is one other
machine on the local network (brian), whose eth0 is at the other end of the
crossed cable.

I used to have shorewall configured so that connections from brian were
correctly masqueraded to the Internet through trevor.  Then one day it
stopped working: both a power cut and a Debian upgrade of shorewall on
trevor seemed to coincide with the time it happened.

Now, commands like ''ping 216.239.39.99'' (that''s
google.com) from brian have
100% packet loss.  Connections using a web browser (for example) also time
out.

I have such a simple setup it is hard for me to see what could possibly have
gone wrong, so I''m now asking for help here after spending a good few
hours
looking for answers (and learning a lot about shorewall and packet filtering
in the process!).

My inexperienced eye, looking at the status logs, seems to think that the
pings get as far as the two PREROUTING chains (Mangle and Nat) as described
in http://shorewall.net/NetfilterOverview.html.  Then it appears they get
lost somewhere around the "routing decision" because there
aren''t any other
packets recorded in the status log for any other chains.

Here are my details:

trevor:/etc/shorewall# shorewall version
2.0.8

trevor:/etc/shorewall# ip addr show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,NOTRAILERS,UP> mtu 1500 qdisc pfifo_fast
qlen
1000
    link/ether 00:c0:f0:4b:58:aa brd ff:ff:ff:ff:ff:ff
    inet 80.6.126.110/24 brd 255.255.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:50:bf:ec:86:46 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global eth1

trevor:/etc/shorewall# ip route show
192.168.0.0/24 dev eth1  proto kernel  scope link  src 192.168.0.1 
80.6.126.0/24 dev eth0  proto kernel  scope link  src 80.6.126.110 
default via 80.6.126.254 dev eth0 

trevor:/etc/shorewall# shorewall reset
Loading /usr/share/shorewall/functions...
Processing /etc/shorewall/params ...
Processing /etc/shorewall/shorewall.conf...
Loading Modules...
Shorewall Counters Reset

(now we switch to brian, the local machine)

brian:~# ping 216.239.39.99
PING 216.239.39.99 (216.239.39.99): 56 data bytes

--- 216.239.39.99 ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss
brian:~#

(back to trevor)

trevor:/etc/shorewall# shorewall status > /tmp/status.txt
RTNETLINK answers: Invalid argument
Dump terminated
RTNETLINK answers: Invalid argument
Dump terminated

trevor:/etc/shorewall# gzip /tmp/status.txt 

(/tmp/status.txt.gz attached)


I installed shorewall from Debian woody originally, and am now running
Debian sarge.

The contents of my /etc/shorewall directory is attached as
shorewall-files.txt.gz.  It was created with this script:

--cut--
#!/bin/bash
cd /etc/shorewall
for file in *; do
    echo "========================="
    echo "$file"
    echo "-----------"
    grep -v "^\(#\|$\)" $file
    echo
done
--cut--


Connections between trevor and brian are fine.  On brian I can run ''X
-query
trevor'' and get an X session with full connectivity to the Internet,
because
all the connections really come from trevor.  Connectivity between trevor
and the Internet seems fine.

It is worth mentioning that the behaviour is the same whether brian is
running Debian or Windows, so I think it is not likely to be a problem with
brian''s configuration.

I am not subscribed to this list, so please Cc me on any replies.

Thanks in advance for any guidance you may be able to send my way.

Regards,
Paul Stephenson

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Stephenson wrote:

|
| trevor:/etc/shorewall# shorewall status > /tmp/status.txt
| RTNETLINK answers: Invalid argument
| Dump terminated
| RTNETLINK answers: Invalid argument
| Dump terminated

Something is wrong with your routing -- the above error messages are
occuring at about the point where /sbin/shorewall appends the routing
information to the output.

What does the output of "ip route ls" look like?

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBWHdOO/MAbZfjDLIRAlMJAJ0Xni5GPtkTGbGw0yzQUQ9e7TN+ygCgi5HD
/HgzYovGQH5+qyRjq4c6iRc=a2WK
-----END PGP SIGNATURE-----

On Mon, Sep 27, 2004 at 01:25:50PM -0700, Tom Eastep
wrote:
> Paul Stephenson wrote:
> 
> |
> | trevor:/etc/shorewall# shorewall status > /tmp/status.txt
> | RTNETLINK answers: Invalid argument
> | Dump terminated
> | RTNETLINK answers: Invalid argument
> | Dump terminated
> 
> Something is wrong with your routing -- the above error messages are
> occuring at about the point where /sbin/shorewall appends the routing
> information to the output.
> 
> What does the output of "ip route ls" look like?
Here it is:

trevor:/etc/shorewall# ip route ls
192.168.0.0/24 dev eth1  proto kernel  scope link  src 192.168.0.1 
80.6.126.0/24 dev eth0  proto kernel  scope link  src 80.6.126.110 
default via 80.6.126.254 dev eth0 

Thanks for your quick reply!

Paul

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Stephenson wrote:
| Hello,
|
| I have a pretty standard two-interface setup with masquerading, so the
local
| network can connect through the firewall to the Internet.
|
| On the firewall box (trevor), eth0 is connected to a cable modem and
eth1 is
| connected to the local network via a crossed cable.  There is one other
| machine on the local network (brian), whose eth0 is at the other end
of the
| crossed cable.
|
| I used to have shorewall configured so that connections from brian were
| correctly masqueraded to the Internet through trevor.  Then one day it
| stopped working: both a power cut and a Debian upgrade of shorewall on
| trevor seemed to coincide with the time it happened.
|
| Now, commands like ''ping 216.239.39.99'' (that''s
google.com) from brian
have
| 100% packet loss.  Connections using a web browser (for example) also time
| out.
|

After a "shorewall clear", can brian and trevor communicate ok?

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBWHz9O/MAbZfjDLIRAj9gAKDJMCznA0uvo3YmJPEkT2m6c4t2rACeLxm8
WH3ii6yWZQY2k2OUUWLHmPU=anl9
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Eastep wrote:
| Paul Stephenson wrote:
| | Hello,
| |
| | I have a pretty standard two-interface setup with masquerading, so the
| local
| | network can connect through the firewall to the Internet.
| |
| | On the firewall box (trevor), eth0 is connected to a cable modem and
| eth1 is
| | connected to the local network via a crossed cable.  There is one other
| | machine on the local network (brian), whose eth0 is at the other end
| of the
| | crossed cable.
| |
| | I used to have shorewall configured so that connections from brian were
| | correctly masqueraded to the Internet through trevor.  Then one day it
| | stopped working: both a power cut and a Debian upgrade of shorewall on
| | trevor seemed to coincide with the time it happened.
| |
| | Now, commands like ''ping 216.239.39.99'' (that''s
google.com) from brian
| have
| | 100% packet loss.  Connections using a web browser (for example) also
| time
| | out.
| |
|
| After a "shorewall clear", can brian and trevor communicate ok?
|

I notice that you have IP_FORWARDING=Keep in your shorewall.conf file so
unless something external to Shorewall is turning it on, then you have
forwarding disabled on your firewall. If ''cat
/proc/sys/net/ipv4/ip_forward'' returns ''0'' then set
IP_FORWARDING=On and
restart Shorewall.

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBWIYbO/MAbZfjDLIRAhHhAJ95xlo0m51l5kg+FV6HcX9iofHQZwCgyLFR
jsPIL05ITQcON6pq8GRhiXI=H59Y
-----END PGP SIGNATURE-----

On Mon, Sep 27, 2004 at 01:51:25PM -0700, Tom Eastep
wrote:
> Please copy the list on your reply.
> 
> Paul Stephenson wrote:
> | Hello,
> |
> | I have a pretty standard two-interface setup with masquerading, so the
> local
> | network can connect through the firewall to the Internet.
> |
> | On the firewall box (trevor), eth0 is connected to a cable modem and
> eth1 is
> | connected to the local network via a crossed cable.  There is one other
> | machine on the local network (brian), whose eth0 is at the other end
> of the
> | crossed cable.
> |
> | I used to have shorewall configured so that connections from brian were
> | correctly masqueraded to the Internet through trevor.  Then one day it
> | stopped working: both a power cut and a Debian upgrade of shorewall on
> | trevor seemed to coincide with the time it happened.
> |
> | Now, commands like ''ping 216.239.39.99''
(that''s google.com) from brian
> have
> | 100% packet loss.  Connections using a web browser (for example) also
time
> | out.
> |
> 
> After a "shorewall clear", can brian and trevor communicate ok?
Yes: with each other, they can communicate as well as ever.  trevor can still
access the Internet.

I have now looked a little further into this error message, since it seems
relevant:

trevor:/etc/shorewall# ip rule show
RTNETLINK answers: Invalid argument
Dump terminated

I found this on RedHat''s bugzilla site:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=7004

They seem to decide that the resolution is setting CONFIG_IP_ADVANCED_ROUTER
in the kernel configuration.  I notice that I don''t have that compiled
into
my kernel.

The help for CONFIG_IP_ADVANCED_ROUTER says that it doesn''t have any
effect
on the kernel itself; just that it won''t ask some following questions. 
But
maybe there is something else I should have turned on in my kernel?

The help also says this:

---
  If you turn on IP forwarding, you will also get the rp_filter, which
  automatically rejects incoming packets if the routing table entry
  for their source address doesn''t match the network interface
they''re
  arriving on. This has security advantages because it prevents the
  so-called IP spoofing, however it can pose problems if you use
  asymmetric routing (packets from you to a host take a different path
  than packets from that host to you) or if you operate a non-routing
  host which has several IP addresses on different interfaces. To turn
  rp_filter off use:

        echo 0 > /proc/sys/net/ipv4/conf/<device>/rp_filter
  or
        echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
---

Well, I thought I''d try the last line, and guess what?  All seems to
work
again!

My next question is this: what _should_ I have done?  Is that line the
"correct" way to resolve my problem or might there be something else I
have
missed.  Perhaps my routing table is slightly broken in some way?

Perhaps these questions are not for this list, as it doesn''t seem to be
an
issue for shorewall at all.

Many thanks Tom for your pointers, especially picking up on that RTNETLINK
error message.

Paul

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Stephenson wrote:
| On Mon, Sep 27, 2004 at 01:51:25PM -0700, Tom Eastep wrote:
|
|>Please copy the list on your reply.
|>
|>Paul Stephenson wrote:
|>| Hello,
|>|
|>| I have a pretty standard two-interface setup with masquerading, so the
|>local
|>| network can connect through the firewall to the Internet.
|>|
|>| On the firewall box (trevor), eth0 is connected to a cable modem and
|>eth1 is
|>| connected to the local network via a crossed cable.  There is one other
|>| machine on the local network (brian), whose eth0 is at the other end
|>of the
|>| crossed cable.
|>|
|>| I used to have shorewall configured so that connections from brian were
|>| correctly masqueraded to the Internet through trevor.  Then one day it
|>| stopped working: both a power cut and a Debian upgrade of shorewall on
|>| trevor seemed to coincide with the time it happened.
|>|
|>| Now, commands like ''ping 216.239.39.99''
(that''s google.com) from brian
|>have
|>| 100% packet loss.  Connections using a web browser (for example)
also time
|>| out.
|>|
|>
|>After a "shorewall clear", can brian and trevor communicate ok?
|
|
| Yes: with each other, they can communicate as well as ever.  trevor
can still
| access the Internet.
|
| I have now looked a little further into this error message, since it seems
| relevant:
|
| trevor:/etc/shorewall# ip rule show
| RTNETLINK answers: Invalid argument
| Dump terminated
|
| I found this on RedHat''s bugzilla site:
| https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=7004
|
| They seem to decide that the resolution is setting
CONFIG_IP_ADVANCED_ROUTER
| in the kernel configuration.  I notice that I don''t have that
compiled
into
| my kernel.
|
| The help for CONFIG_IP_ADVANCED_ROUTER says that it doesn''t have any
effect
| on the kernel itself; just that it won''t ask some following
questions.
~ But
| maybe there is something else I should have turned on in my kernel?
|
| The help also says this:
|
| ---
|   If you turn on IP forwarding, you will also get the rp_filter, which
|   automatically rejects incoming packets if the routing table entry
|   for their source address doesn''t match the network interface
they''re
|   arriving on. This has security advantages because it prevents the
|   so-called IP spoofing, however it can pose problems if you use
|   asymmetric routing (packets from you to a host take a different path
|   than packets from that host to you) or if you operate a non-routing
|   host which has several IP addresses on different interfaces. To turn
|   rp_filter off use:
|
|         echo 0 > /proc/sys/net/ipv4/conf/<device>/rp_filter
|   or
|         echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
| ---
|
| Well, I thought I''d try the last line, and guess what?  All seems to
work
| again!
|
| My next question is this: what _should_ I have done?

Remove ''routefilter'' from your /etc/shorewall/interfaces file.

| Is that line the
| "correct" way to resolve my problem or might there be something else
I
have
| missed.  Perhaps my routing table is slightly broken in some way?
|

There is still something wrong -- your setup should work fine with route
filtering enabled.

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBWIgVO/MAbZfjDLIRAoQpAJ4nE8Qopubi/uew8G8ZT4ZO7bU3KgCcDFxp
CyzaYLP8pHEZ8Shz/Mb8nhY=SHPY
-----END PGP SIGNATURE-----

On Mon, Sep 27, 2004 at 02:29:00PM -0700, Tom Eastep
wrote:
> Tom Eastep wrote:
> | Paul Stephenson wrote:
> | | Hello,
> | |
> | | I have a pretty standard two-interface setup with masquerading, so the
> | local
> | | network can connect through the firewall to the Internet.
> | |
> | | Now, commands like ''ping 216.239.39.99''
(that''s google.com) from brian
> | have
> | | 100% packet loss.  Connections using a web browser (for example) also
> | time
> | | out.
> | |
> |
> | After a "shorewall clear", can brian and trevor communicate ok?
> |
> 
> I notice that you have IP_FORWARDING=Keep in your shorewall.conf file so
> unless something external to Shorewall is turning it on, then you have
> forwarding disabled on your firewall. If ''cat
> /proc/sys/net/ipv4/ip_forward'' returns ''0'' then
set IP_FORWARDING=On and
> restart Shorewall.
ip_forward seems to be on already:

trevor:/# cat /proc/sys/net/ipv4/ip_forward 
1

I''ll maybe try fiddling with some of these settings tomorrow to see if
I can
make more sense of what''s going on.

Thanks for all your help,
Paul

On Mon, Sep 27, 2004 at 10:41:47PM +0100, paul wrote:
> On Mon, Sep 27, 2004 at 02:29:00PM -0700, Tom Eastep wrote:
> > Tom Eastep wrote:
> > | Paul Stephenson wrote:
> > | | Hello,
> > | |
> > | | I have a pretty standard two-interface setup with masquerading, so
the
> > | local
> > | | network can connect through the firewall to the Internet.
> > | |
> > | | Now, commands like ''ping 216.239.39.99''
(that''s google.com) from brian
> > | have
> > | | 100% packet loss.  Connections using a web browser (for example)
also
> > | time
> > | | out.
> > | |
> > |
> > | After a "shorewall clear", can brian and trevor
communicate ok?
> > |
> > 
> > I notice that you have IP_FORWARDING=Keep in your shorewall.conf file
so
> > unless something external to Shorewall is turning it on, then you have
> > forwarding disabled on your firewall. If ''cat
> > /proc/sys/net/ipv4/ip_forward'' returns ''0''
then set IP_FORWARDING=On and
> > restart Shorewall.
> 
> ip_forward seems to be on already:
> 
> trevor:/# cat /proc/sys/net/ipv4/ip_forward 
> 1
> 
> I''ll maybe try fiddling with some of these settings tomorrow to
see if I can
> make more sense of what''s going on.
Well, this evening I thought I''d try rebooting the firewall (trevor) to
see
what state it came up in.  The connections from brian to the Internet
stopped working again, as I''d expected.  However, what I
didn''t expect was
that `cat /proc/sys/net/ipv4/ip_forward` was now set to ''0''. 
I set
IP_FORWARDING=On in my shorewall.conf file as you suggested above and all
now works (and also after a reboot).

There is still something suspicious about ''ip rule ls''
returning the
RTNETLINK error, but it now looks like it may be unrelated.  Perhaps I do
need some extra kernel options defined, but maybe I don''t need the
features
anyway.

I am also wondering, if ip_forward is ''0'' after a reboot, what
set it to ''1''
yesterday (or before).

Looking through some old copies of shorewall.conf I have lying around, such
as the one before my upgrade to Debian sarge, it seems that the default Debian
config file has indeed changed from

	IP_FORWARDING="On"
to

	IP_FORWARDING=Keep

so maybe I didn''t take enough care when setting shorewall up again
after the
upgrade.

Thanks again Tom.  My 2-year-old daughter can now have "her" computer
back
to play games on the BBC CBeebies website!

Paul

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Stephenson wrote:
> Well, this evening I thought I''d try rebooting the firewall
(trevor)
to see
> what state it came up in.  The connections from brian to the Internet
> stopped working again, as I''d expected.  However, what I
didn''t expect was
> that `cat /proc/sys/net/ipv4/ip_forward` was now set to
''0''.  I set
> IP_FORWARDING=On in my shorewall.conf file as you suggested above and all
> now works (and also after a reboot).
I have always released shorewall.conf with IP_FORWARDING=On. The Debian
team prefer conservative defaults so they prevailed upon the Debian
Shorewall maintainer to change the default to IP_FORWARDING=Keep.
>
> There is still something suspicious about ''ip rule ls''
returning the
> RTNETLINK error, but it now looks like it may be unrelated.  Perhaps I do
> need some extra kernel options defined, but maybe I don''t need the
features
> anyway.
I need to make the /sbin/shorewall script more tolerant of ''ip rule
ls''
failing though.
>
> I am also wondering, if ip_forward is ''0'' after a reboot,
what set it
to ''1''
> yesterday (or before).
>
> Looking through some old copies of shorewall.conf I have lying around,
such
> as the one before my upgrade to Debian sarge, it seems that the
default Debian
> config file has indeed changed from
>
> 	IP_FORWARDING="On"
> to
>
> 	IP_FORWARDING=Keep
>
> so maybe I didn''t take enough care when setting shorewall up again
after the
> upgrade.
>
See above.

- -Tom
- --
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBWcUyO/MAbZfjDLIRAnrmAJ9FkwRN1fSKd3WbkocIybJxiVxi8wCgoVx7
nsU8q5OXZMfwvxC55mlVEZ8=byyF
-----END PGP SIGNATURE-----