Shorewall users - Feb 2005 - SW 2.2.0: 4 interface system, log reports impossible "IN=" and DROPS

This one is really throwing me.  Thanks in advance for
any advice.

I''m working on a 4 port firewall system.  It is
running heartbeat+drbd.

Primary box looks like this:
eth0 -> net/cicso router
        192.168.144.2/29
eth1 -> drbd/heartbeat crossover cable
        192.168.254.253/30
eth2 -> dmz
        192.168.144.10/24
eth3 -> loc
        192.168.101.2/24

The IP''s of the secondary box are one number higher. 
The hardware is the same, and the OS was imaged from
the primary machine.

So when I try to ping from the secondary box to the
primary, the primary spits out kernel log rejecting on
smurfs.  I edit interfaces to remove nosmurfs from
eth3, restart.  Then it rejects on mac_list, so I
remove that and restart.  THEN I GET THIS showing that
it has been dropped on IN=eth0, while previous logs
all properly showed eth3:

Feb  3 21:28:38 barbrady kernel:
Shorewall:net2all:DROP:IN=eth0 OUTMAC=00:04:75:eb:a9:00:00:50:8b:e8:74:9c:08:00
SRC=192.168.101.3 DST=192.168.101.2 LEN=84 TOS=0x00
PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0
ID=29738 SEQ=0
Feb  3 21:28:39 barbrady kernel:
Shorewall:net2all:DROP:IN=eth0 OUTMAC=00:04:75:eb:a9:00:00:50:8b:e8:74:9c:08:00
SRC=192.168.101.3 DST=192.168.101.2 LEN=84 TOS=0x00
PREC=0x00 TTL=64 ID=1 DF PROTO=ICMP TYPE=8 CODE=0
ID=29738 SEQ=1

But when I do a tcpdump:
shorewall]# tcpdump -i eth3 -n
tcpdump: verbose output suppressed, use -v or -vv for
full protocol decode
listening on eth3, link-type EN10MB (Ethernet),
capture size 96 bytes
21:18:51.103139 IP 192.168.101.3 > 192.168.101.2: icmp
64: echo request seq 0
21:18:52.102248 IP 192.168.101.3 > 192.168.101.2: icmp
64: echo request seq 1
21:18:56.101481 arp who-has 192.168.101.2 tell
192.168.101.3
21:18:56.101513 arp reply 192.168.101.2 is-at
00:50:8b:e8:47:64

4 packets captured
4 packets received by filter
0 packets dropped by kernel


These packets ARE coming in on eth3, but I have no
idea why they would be rejected as smurf packets, when
they aren''t on broadcast, or macblacklist when I
haven''t touched that.  Then the jump to an IN=eth0
threw me more than anything.

The masq file is completely commented out, so I have
no idea how in the world this is happening.  I''ve
tried restarting the box to no avail.

Thanks in advance.

Brian Neu wrote:
> This one is really throwing me.  Thanks in advance for
> any advice.
> 
>
Brian -- this isn''t a problem report; if anything, it is the opening
chapter of a very bad novel.

Please visit http://shorewall.net/support.htm and review the guidelines
for posting a problem report then try again. In particular, please pay
attention to the section beginning THIS IS IMPORTANT!

Thanks,
-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

Yes, welcome to my bad novel :).  Sorry for the bad
format.  I''ve had to R more FM''s in the past two days
than my brain could take.

#shorewall version output is 2.2.0 .  I first
installed RC5, then installed the release.  

Files are attached.

--- Tom Eastep <teastep@shorewall.net> wrote:
> Brian Neu wrote:
> > This one is really throwing me.  Thanks in advance
> for
> > any advice.
> > 
> >
> 
> Brian -- this isn''t a problem report; if anything,
> it is the opening
> chapter of a very bad novel.
> 
> Please visit http://shorewall.net/support.htm and
> review the guidelines
> for posting a problem report then try again. In
> particular, please pay
> attention to the section beginning THIS IS
> IMPORTANT!
> 
> Thanks,
> -Tom
> -- 
> Tom Eastep    \ Nothing is foolproof to a
> sufficiently talented fool
> Shoreline,     \ http://shorewall.net
> Washington USA  \ teastep@shorewall.net
> PGP Public Key   \
> https://lists.shorewall.net/teastep.pgp.key
> _______________________________________________
> Shorewall-users mailing list
> Post: Shorewall-users@lists.shorewall.net
> Subscribe/Unsubscribe:
>
https://lists.shorewall.net/mailman/listinfo/shorewall-users
> Support: http://www.shorewall.net/support.htm
> FAQ: http://www.shorewall.net/FAQ.htm
>

Brian Neu wrote:
> Yes, welcome to my bad novel :).
:-)
> Sorry for the bad
> format.  I''ve had to R more FM''s in the past two days
> than my brain could take.
> 
> #shorewall version output is 2.2.0 .  I first
> installed RC5, then installed the release.  
> 
> Files are attached.
Brian -- what does your /etc/shorewall/interfaces file look like?

Thanks,
-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

Brian Neu wrote:
> 
> So when I try to ping from the secondary box to the
> primary, the primary spits out kernel log rejecting on
> smurfs.  I edit interfaces to remove nosmurfs from
> eth3, restart.  Then it rejects on mac_list, so I
> remove that and restart.  THEN I GET THIS showing that
> it has been dropped on IN=eth0, while previous logs
> all properly showed eth3:
Are eth0 and eth3 connected to the same hub/switch?

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

interfaces:
net     eth0            192.168.144.7/29       
tcpflags,blacklist,nobogons
drbd    eth1            192.168.254.255/30
dmz     eth2            192.168.144.255/24     
tcpflags,newnotsyn,nosmurfs,nobogons
loc     eth3            192.168.101.255/24     
tcpflags,nobogons,dhcp

The answer about the same switch is yes/no.  It is the
same switch, but they are on separate VLANs.

And remember, it''s coming in on eth3 -- I can watch it
with tcpdump as it comes in.  I can also verify that
the hardware address is the hardware address of eth3
on the other machine.  But somehow, it gets recorded
in syslog as "IN=eth0" and it gets DROPped for rules
of eth0.   

I know I''m doing something a little hokey by splitting
one of the subnets, but because routing tables put the
smallest networks at the top of the routing tables, it
works.

Thanks again for looking at this.

--- Tom Eastep <teastep@shorewall.net> wrote:
> Brian Neu wrote:
> 
> > 
> > So when I try to ping from the secondary box to
> the
> > primary, the primary spits out kernel log
> rejecting on
> > smurfs.  I edit interfaces to remove nosmurfs from
> > eth3, restart.  Then it rejects on mac_list, so I
> > remove that and restart.  THEN I GET THIS showing
> that
> > it has been dropped on IN=eth0, while previous
> logs
> > all properly showed eth3:
> 
> Are eth0 and eth3 connected to the same hub/switch?
> 
> -Tom
> -- 
> Tom Eastep    \ Nothing is foolproof to a
> sufficiently talented fool
> Shoreline,     \ http://shorewall.net
> Washington USA  \ teastep@shorewall.net
> PGP Public Key   \
> https://lists.shorewall.net/teastep.pgp.key
> _______________________________________________
> Shorewall-users mailing list
> Post: Shorewall-users@lists.shorewall.net
> Subscribe/Unsubscribe:
>
https://lists.shorewall.net/mailman/listinfo/shorewall-users
> Support: http://www.shorewall.net/support.htm
> FAQ: http://www.shorewall.net/FAQ.htm
>

Brian Neu wrote:
> interfaces:
> net     eth0            192.168.144.7/29       
> tcpflags,blacklist,nobogons
> drbd    eth1            192.168.254.255/30
> dmz     eth2            192.168.144.255/24     
> tcpflags,newnotsyn,nosmurfs,nobogons
> loc     eth3            192.168.101.255/24     
> tcpflags,nobogons,dhcp
Brian -- read again what the third column is supposed to contain (Hint;
it''s labeled BROADCAST, not NETWORK).
> 
> The answer about the same switch is yes/no.  It is the
> same switch, but they are on separate VLANs.
> 
> And remember, it''s coming in on eth3 -- I can watch it
> with tcpdump as it comes in.  I can also verify that
> the hardware address is the hardware address of eth3
> on the other machine.  But somehow, it gets recorded
> in syslog as "IN=eth0" and it gets DROPped for rules
> of eth0.   
That means that eth0 is receiving ARP "who-has" requests that it
shouldn''t; it''s as simple as that -- if you set arp_filter on
all
interfaces (/etc/shorewall/intefaces), I suspect that these symptoms
will go away but it sounds like something is wrong with your VLAN setup.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

On Fri, 2005-02-04 at 08:22 -0800, Brian Neu wrote:
> I know I''m doing something a little hokey by splitting
> one of the subnets, but because routing tables put the
> smallest networks at the top of the routing tables, it
> works.
Whether it works or not, I would greatly discourage you from doing
something as hokey.  If you are using non-routeables on those nets
anyway, why not just use completely separate subnets?  If that won''t
work for whatever reason, you would really be better off splitting that
subnet properly (sure, you''ll lose a bunch of addresses, but thats
life)
and keeping things legit.  Whenever you try to ''out-fox'' IP,
it comes
back to bite you in the backside.

-- 
David Hollis <dhollis@davehollis.com>

Tom Eastep wrote:
> That means that eth0 is receiving ARP "who-has" requests that it
> shouldn''t; it''s as simple as that -- if you set
arp_filter on all
> interfaces (/etc/shorewall/intefaces), I suspect that these symptoms
> will go away but it sounds like something is wrong with your VLAN setup.
That is what it looks like to me as well. I''m running about 7 systems 
with a lots of VlANs under Linux at the moment with great success.
We''re
using HP ProCurve switches.

Maybe I can be of assistance?

1. If you are using multiple NICs, you probably want your VLANs to be 
"untagged" for those ports and not send any other VLAN traffic traffic
at all
2. You can use a single NIC and have your Linux box to the untagging. 
This way you end up with eth0.1, eth0.2, eth0.3, etc. (Actually, you can 
name them vlan0, vlan1, etc, too.

Can you describe your setup a bit more?

A.

Tom, you are the Lord of All Things Shorewall.  Your
greatness cannot be grasped be mere mortals or even
lesser gods.  May you live forever and continue to
enlighten those in the perils IP darkness.

Death to he who mislabeled the VLANs on the front of
my  switch.

the end

--- Tom Eastep <teastep@shorewall.net> wrote:
> Brian Neu wrote:
> > interfaces:
> > net     eth0            192.168.144.7/29       
> > tcpflags,blacklist,nobogons
> > drbd    eth1            192.168.254.255/30
> > dmz     eth2            192.168.144.255/24     
> > tcpflags,newnotsyn,nosmurfs,nobogons
> > loc     eth3            192.168.101.255/24     
> > tcpflags,nobogons,dhcp
> 
> Brian -- read again what the third column is
> supposed to contain (Hint;
> it''s labeled BROADCAST, not NETWORK).
> 
> > 
> > The answer about the same switch is yes/no.  It is
> the
> > same switch, but they are on separate VLANs.
> > 
> > And remember, it''s coming in on eth3 -- I can
> watch it
> > with tcpdump as it comes in.  I can also verify
> that
> > the hardware address is the hardware address of
> eth3
> > on the other machine.  But somehow, it gets
> recorded
> > in syslog as "IN=eth0" and it gets DROPped for
> rules
> > of eth0.   
> 
> That means that eth0 is receiving ARP "who-has"
> requests that it
> shouldn''t; it''s as simple as that -- if you set
> arp_filter on all
> interfaces (/etc/shorewall/intefaces), I suspect
> that these symptoms
> will go away but it sounds like something is wrong
> with your VLAN setup.
> 
> -Tom
> -- 
> Tom Eastep    \ Nothing is foolproof to a
> sufficiently talented fool
> Shoreline,     \ http://shorewall.net
> Washington USA  \ teastep@shorewall.net
> PGP Public Key   \
> https://lists.shorewall.net/teastep.pgp.key
> _______________________________________________
> Shorewall-users mailing list
> Post: Shorewall-users@lists.shorewall.net
> Subscribe/Unsubscribe:
>
https://lists.shorewall.net/mailman/listinfo/shorewall-users
> Support: http://www.shorewall.net/support.htm
> FAQ: http://www.shorewall.net/FAQ.htm
>

Brian Neu wrote:
> Tom, you are the Lord of All Things Shorewall.  Your
> greatness cannot be grasped be mere mortals or even
> lesser gods.  May you live forever and continue to
> enlighten those in the perils IP darkness.
The novel gets worse :-)
> 
> Death to he who mislabeled the VLANs on the front of
> my  switch.
> 
Yep. Now if you fix your /etc/shorewall/interfaces file, you should be
all set.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key

net     eth0            192.168.144.7          
tcpflags,blacklist,nobogons,arp_filter
drbd    eth1            192.168.254.255
dmz     eth2            192.168.144.255        
tcpflags,nosmurfs,nobogons,arp_filter
loc     eth3            192.168.101.255        
tcpflags,nosmurfs,nobogons,dhcp,arp_filter

interfaces file is happy again.  Since I''m 500 miles
away from the switch and I have no clue what the IP
is, the arp_filter is going to have to do for this
test env.

I was under the impression that a NIC would only arp
reply their own IP, and not an IP of another NIC the
host had.  It is very clear that the net zone and the
loc zone are on the same VLAN though when I saw arp
requests and replies from the upstream router,
followed by eth0 replying with it''s own MAC for a
request of an IP on eth3.  arp_filter fixed it for
now.  While I see in the interfaces file that
shorewall will behave this way, will a standard Linux
box with fowarding enabled reply to arp requests for
IP''s on other interfaces?

The production environment has physically separated
networks.  This is a testing environment which was
obviously ill configured, along with a few poor
configurations by a recently humbled sysadmin.

A sincere ''thank you'' to all who offered help.


--- Tom Eastep <teastep@shorewall.net> wrote:
> Brian Neu wrote:
> > Tom, you are the Lord of All Things Shorewall. 
> Your
> > greatness cannot be grasped be mere mortals or
> even
> > lesser gods.  May you live forever and continue to
> > enlighten those in the perils IP darkness.
> 
> The novel gets worse :-)
> 
> > 
> > Death to he who mislabeled the VLANs on the front
> of
> > my  switch.
> > 
> 
> Yep. Now if you fix your /etc/shorewall/interfaces
> file, you should be
> all set.
> 
> -Tom
> -- 
> Tom Eastep    \ Nothing is foolproof to a
> sufficiently talented fool
> Shoreline,     \ http://shorewall.net
> Washington USA  \ teastep@shorewall.net
> PGP Public Key   \
> https://lists.shorewall.net/teastep.pgp.key
> _______________________________________________
> Shorewall-users mailing list
> Post: Shorewall-users@lists.shorewall.net
> Subscribe/Unsubscribe:
>
https://lists.shorewall.net/mailman/listinfo/shorewall-users
> Support: http://www.shorewall.net/support.htm
> FAQ: http://www.shorewall.net/FAQ.htm
>

Brian Neu wrote:
> net     eth0            192.168.144.7          
> tcpflags,blacklist,nobogons,arp_filter
> drbd    eth1            192.168.254.255
> dmz     eth2            192.168.144.255        
> tcpflags,nosmurfs,nobogons,arp_filter
> loc     eth3            192.168.101.255        
> tcpflags,nosmurfs,nobogons,dhcp,arp_filter
> 
> interfaces file is happy again.  Since I''m 500 miles
> away from the switch and I have no clue what the IP
> is, the arp_filter is going to have to do for this
> test env.
> 
> I was under the impression that a NIC would only arp
> reply their own IP, and not an IP of another NIC the
> host had.  It is very clear that the net zone and the
> loc zone are on the same VLAN though when I saw arp
> requests and replies from the upstream router,
> followed by eth0 replying with it''s own MAC for a
> request of an IP on eth3.  arp_filter fixed it for
> now.  While I see in the interfaces file that
> shorewall will behave this way, will a standard Linux
> box with fowarding enabled reply to arp requests for
> IP''s on other interfaces?
> 
Yes -- that behavior has nothing to do with Shorewall. It might be
useful for you to review the Introduction article on the Shorewall
website so you are clear about what Shorewall is (and isn''t).
> The production environment has physically separated
> networks.  This is a testing environment which was
> obviously ill configured, along with a few poor
> configurations by a recently humbled sysadmin.
> 
> A sincere ''thank you'' to all who offered help.
You''re welcome.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key