Shorewall 4.5.6, CentOS 6.3. The shorewall box runs several KVM virtual
machines, each of which has an interface attached to the two
"internal"
bridges br1 and br2 (and each thus has two IP''s; one on 192.168.0.0/22
and
one on 192.168.4.0/22):
Internet ("net")
^
|
+---------------+
| br0 |
net1=192.168.4.0/22 | | net2=192.168.0.0/22
-----------------------+ br1 br2 +---------------------------------
| | | |
| | | |
Client A +---------------+ Client B
(hosts KVM1, KVM2, etc)
There are no shorewall rules that prohibit net1 <-> net2 traffic. I have:
/etc/shorewall/interfaces:
net br0 -
tcpflags,routefilter,nosmurfs,logmartians,blacklist,arp_filter
net1 br1 - tcpflags,nosmurfs,maclist
net2 br2 - tcpflags,nosmurfs,maclist
/etc/shorewall/zones:
fw firewall
net ipv4
net1 ipv4
net2 ipv4
/etc/shorewall/policy:
net1 net ACCEPT
net1 net2 ACCEPT
net2 net ACCEPT
net2 net1 ACCEPT
net all DROP info
all all REJECT info
Client A can ldapsearch (for example) to a KVM machine by IP address by
using the KVM machine''s net1 IP address. Client B can likewise
communicate
using the KVM machine''s net2 IP address. However, neither client can
communicate by using the address on the opposing segment (eg, Client A
using KVM1_net2_IP); the packets are rejected by shorewall with the
appropriate log message:
Sep 8 16:13:05 gw-2 kernel: Shorewall:br1_rec:REJECT:IN=br1 OUT=br2
SRC=192.168.5.204 DST=192.168.0.14 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=10716
DF PROTO=TCP SPT=55536 DPT=389 WINDOW=65535 RES=0x00 SYN URGP=0
I would have expected the "net1 net2 ACCEPT" and "net2 net1
ACCEPT"
policies to allow this, since the KVM machine is logically attached to
both networks. Since it doesn''t work, I am missing something. I have
tried
using the bridge option in the interfaces file to no effect. I''d
appreciate it if someone can give me a clue.
TIA,
Steve
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today''s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 2:06 PM, Steve Thompson wrote:> > Shorewall 4.5.6, CentOS 6.3. The shorewall box runs several KVM virtual > machines, each of which has an interface attached to the two "internal" > bridges br1 and br2 (and each thus has two IP''s; one on 192.168.0.0/22 and > one on 192.168.4.0/22): > > Internet ("net") > ^ > | > +---------------+ > | br0 | > net1=192.168.4.0/22 | | net2=192.168.0.0/22 > -----------------------+ br1 br2 +--------------------------------- > | | | | > | | | | > Client A +---------------+ Client B > (hosts KVM1, KVM2, etc) > > There are no shorewall rules that prohibit net1 <-> net2 traffic. I have: > > /etc/shorewall/interfaces: > net br0 - tcpflags,routefilter,nosmurfs,logmartians,blacklist,arp_filter > net1 br1 - tcpflags,nosmurfs,maclist > net2 br2 - tcpflags,nosmurfs,maclist > > /etc/shorewall/zones: > fw firewall > net ipv4 > net1 ipv4 > net2 ipv4 > > /etc/shorewall/policy: > net1 net ACCEPT > net1 net2 ACCEPT > net2 net ACCEPT > net2 net1 ACCEPT > net all DROP info > all all REJECT info > > Client A can ldapsearch (for example) to a KVM machine by IP address by > using the KVM machine''s net1 IP address. Client B can likewise communicate > using the KVM machine''s net2 IP address. However, neither client can > communicate by using the address on the opposing segment (eg, Client A > using KVM1_net2_IP); the packets are rejected by shorewall with the > appropriate log message: > > Sep 8 16:13:05 gw-2 kernel: Shorewall:br1_rec:REJECT:IN=br1 OUT=br2 > SRC=192.168.5.204 DST=192.168.0.14 LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=10716 > DF PROTO=TCP SPT=55536 DPT=389 WINDOW=65535 RES=0x00 SYN URGP=0 > > I would have expected the "net1 net2 ACCEPT" and "net2 net1 ACCEPT" > policies to allow this, since the KVM machine is logically attached to > both networks. Since it doesn''t work, I am missing something. I have tried > using the bridge option in the interfaces file to no effect. I''d > appreciate it if someone can give me a clue.Shorewall FAQ 17 and look for <interface>_rec -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sat, 8 Sep 2012, Tom Eastep wrote:> On 9/8/12 2:06 PM, Steve Thompson wrote: >> I would have expected the "net1 net2 ACCEPT" and "net2 net1 ACCEPT" >> policies to allow this, since the KVM machine is logically attached to >> both networks. Since it doesn''t work, I am missing something. I have tried >> using the bridge option in the interfaces file to no effect. I''d >> appreciate it if someone can give me a clue. > > Shorewall FAQ 17 and look for <interface>_recThanks for the pointer, but I don''t think that it helps me (or I just don''t understand how it helps me). I do use the maclist option on the firewall''s interfaces, and I do have a maclist file that lists all relevant MAC addresses, but even if I remove the maclist option from all of the network interfaces altogether, it still doesn''t change anything. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 3:38 PM, Steve Thompson wrote:> On Sat, 8 Sep 2012, Tom Eastep wrote: > >> On 9/8/12 2:06 PM, Steve Thompson wrote: >>> I would have expected the "net1 net2 ACCEPT" and "net2 net1 ACCEPT" >>> policies to allow this, since the KVM machine is logically attached to >>> both networks. Since it doesn''t work, I am missing something. I have tried >>> using the bridge option in the interfaces file to no effect. I''d >>> appreciate it if someone can give me a clue. >> >> Shorewall FAQ 17 and look for <interface>_rec > > Thanks for the pointer, but I don''t think that it helps me (or I just > don''t understand how it helps me). I do use the maclist option on the > firewall''s interfaces, and I do have a maclist file that lists all > relevant MAC addresses, but even if I remove the maclist option from all > of the network interfaces altogether, it still doesn''t change anything. >I don''t believe that. If you remove the ''maclist'' option altogether, then there will be no no br1_rec chain. So the log message that you posted cannot possibly be issued. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 3:38 PM, Steve Thompson wrote:> > Thanks for the pointer, but I don''t think that it helps me (or I just > don''t understand how it helps me). I do use the maclist option on the > firewall''s interfaces, and I do have a maclist file that lists all > relevant MAC addresses, but even if I remove the maclist option from all > of the network interfaces altogether, it still doesn''t change anything. >If you can''t get it sorted out, please post the output of ''shorewall dump'' as a compressed attachment. Thanks, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sat, 8 Sep 2012, Tom Eastep wrote:> I don''t believe that. If you remove the ''maclist'' option altogether, > then there will be no no br1_rec chain. So the log message that you > posted cannot possibly be issued.Indeed the log message is not issued. But no packets pass across the firewall from one bridged interface to the client IP that is connected to the other bridge. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sat, 8 Sep 2012, Tom Eastep wrote:> If you can''t get it sorted out, please post the output of ''shorewall > dump'' as a compressed attachment.Thanks Tom. The basic configuration has been in operation for over 5 years with no problems. It is only when I introduce a virtual machine on the firewall system, and desire to communicate with it using any of its IP addresses, that this problem occurs. I''ve attached the dump, since I am stuck. Steve , ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 4:16 PM, Steve Thompson wrote:> On Sat, 8 Sep 2012, Tom Eastep wrote: > >> If you can''t get it sorted out, please post the output of ''shorewall >> dump'' as a compressed attachment. > > Thanks Tom. The basic configuration has been in operation for over 5 > years with no problems. It is only when I introduce a virtual machine on > the firewall system, and desire to communicate with it using any of its > IP addresses, that this problem occurs. I''ve attached the dump, since I > am stuck. >Here''s a clue. We see this log message: Sep 8 19:14:37 br2_rec:REJECT:IN=br2 OUT=br1 SRC=192.168.0.23 DST=192.168.4.2 LEN=76 TOS=0x00 PREC=0x00 TTL=127 ID=6379 PROTO=UDP SPT=123 DPT=123 LEN=56 It is being rejected because there is maclist no entry for 192.168.0.3 on br2 (which is the bridge on which this packet was received). In fact, there is no maclist entry for that IP address at all. When you see packets being rejected in one of the _rec chains, you must check your maclist entries. Let''s try to solve the other problem where no ''maclist'' entries are present. Please: 1. Remove ''maclist'' from all interfaces. 2. Restart the firewall. 3. Try a connection that fails. 4. Capture and post the output of ''shorewall dump''. -Tom PS -- What in the world are you trying to accomplish with this configuration? I''ve never seen anything so bizarre (dozens of RFC1918 addresses being DNATed to public IPs). Surely there has to be a better way. -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sat, 8 Sep 2012, Tom Eastep wrote:> Here''s a clue. We see this log message: > > Sep 8 19:14:37 br2_rec:REJECT:IN=br2 OUT=br1 SRC=192.168.0.23 > DST=192.168.4.2 LEN=76 TOS=0x00 PREC=0x00 TTL=127 ID=6379 PROTO=UDP > SPT=123 DPT=123 LEN=56 > > It is being rejected because there is maclist no entry for 192.168.0.3 > on br2 (which is the bridge on which this packet was received). > > In fact, there is no maclist entry for that IP address at all. When you > see packets being rejected in one of the _rec chains, you must check > your maclist entries.Ah, but there _is_ an entry in the maclist file for this IP address. An extract: ACCEPT br2 84:2B:2B:47:D6:85 192.168.0.3 ACCEPT br2 84:2B:2B:47:D6:86 192.168.0.3 ACCEPT br2 00:1B:21:6F:2B:54 192.168.0.3 ACCEPT br2 00:1B:21:6F:2B:55 192.168.0.3 which has four entries because the software that builds the maclist file does not know which MAC address is associated with which IP (the machine in question has four interfaces). I know for sure that the maclist is accurate; it has entries for every MAC address on the network (the maclist file has 580 entries). Again, this configuration has worked for 5+ years until the introduction of virtual machines into the configuration.> Let''s try to solve the other problem where no ''maclist'' entries are > present. Please: > 1. Remove ''maclist'' from all interfaces. > 2. Restart the firewall. > 3. Try a connection that fails. > 4. Capture and post the output of ''shorewall dump''.Will do.> PS -- What in the world are you trying to accomplish with this > configuration? I''ve never seen anything so bizarre (dozens of RFC1918 > addresses being DNATed to public IPs). Surely there has to be a better way.Indeed there should be a better way. However, I have this configuration forced on me by "higher" levels. Originally I DNAT''d the whole mess to a single public IP, but if the "higher" levels detect a source of malware from one of the internal systems (Windows, of course), they block the public IP from internet access, hence taking out the entire department. Hence this configuration has essentially been forced on me. I don''t like it one bit, but I''m fighting City Hall. I thank you enormously for your input, Tom. You are a king. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 5:25 PM, Steve Thompson wrote:> On Sat, 8 Sep 2012, Tom Eastep wrote: > >> Here''s a clue. We see this log message: >> >> Sep 8 19:14:37 br2_rec:REJECT:IN=br2 OUT=br1 SRC=192.168.0.23 >> DST=192.168.4.2 LEN=76 TOS=0x00 PREC=0x00 TTL=127 ID=6379 PROTO=UDP >> SPT=123 DPT=123 LEN=56 >> >> It is being rejected because there is maclist no entry for 192.168.0.3 >> on br2 (which is the bridge on which this packet was received). >> >> In fact, there is no maclist entry for that IP address at all. When you >> see packets being rejected in one of the _rec chains, you must check >> your maclist entries. > > Ah, but there _is_ an entry in the maclist file for this IP address. An > extract: > > ACCEPT br2 84:2B:2B:47:D6:85 192.168.0.3 > ACCEPT br2 84:2B:2B:47:D6:86 192.168.0.3 > ACCEPT br2 00:1B:21:6F:2B:54 192.168.0.3 > ACCEPT br2 00:1B:21:6F:2B:55 192.168.0.3 >192.168.0.23 IS NOT 192.168.0.3. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sat, 8 Sep 2012, Tom Eastep wrote:> 192.168.0.23 IS NOT 192.168.0.3.Yes, it isn''t. But 192.168.0.23 is a virtual IP address that is implemented by the keepalived daemon on one of the shorewall system''s virtual machines. It, as such, doesn''t have a MAC address, and isn''t involved in the current situation. In any event, connections to that IP address work. The maclist file is definitely complete as far as I can see. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 5:36 PM, Tom Eastep wrote:> On 9/8/12 5:25 PM, Steve Thompson wrote: >> On Sat, 8 Sep 2012, Tom Eastep wrote: >> >>> Here''s a clue. We see this log message: >>> >>> Sep 8 19:14:37 br2_rec:REJECT:IN=br2 OUT=br1 SRC=192.168.0.23 >>> DST=192.168.4.2 LEN=76 TOS=0x00 PREC=0x00 TTL=127 ID=6379 PROTO=UDP >>> SPT=123 DPT=123 LEN=56 >>> >>> It is being rejected because there is maclist no entry for 192.168.0.3 >>> on br2 (which is the bridge on which this packet was received). >>> >>> In fact, there is no maclist entry for that IP address at all. When you >>> see packets being rejected in one of the _rec chains, you must check >>> your maclist entries. >> >> Ah, but there _is_ an entry in the maclist file for this IP address. An >> extract: >> >> ACCEPT br2 84:2B:2B:47:D6:85 192.168.0.3 >> ACCEPT br2 84:2B:2B:47:D6:86 192.168.0.3 >> ACCEPT br2 00:1B:21:6F:2B:54 192.168.0.3 >> ACCEPT br2 00:1B:21:6F:2B:55 192.168.0.3 >> > > 192.168.0.23 IS NOT 192.168.0.3. >And I apologize for yelling :-) But these problems are all about details. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 5:44 PM, Steve Thompson wrote:> On Sat, 8 Sep 2012, Tom Eastep wrote: > >> 192.168.0.23 IS NOT 192.168.0.3. > > Yes, it isn''t. But 192.168.0.23 is a virtual IP address that is > implemented by the keepalived daemon on one of the shorewall system''s > virtual machines. It, as such, doesn''t have a MAC address, and isn''t > involved in the current situation. In any event, connections to that IP > address work. The maclist file is definitely complete as far as I can see. >The log shows that packets from that IP address are being rejected. That was your initial complaint (the log message in your initial email was similar to the one I posted). If that isn''t a problem, then you must not have any problem. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/8/12 5:44 PM, Steve Thompson wrote:> On Sat, 8 Sep 2012, Tom Eastep wrote: > >> 192.168.0.23 IS NOT 192.168.0.3. > > Yes, it isn''t. But 192.168.0.23 is a virtual IP address that is > implemented by the keepalived daemon on one of the shorewall system''s > virtual machines. It, as such, doesn''t have a MAC address, and isn''t > involved in the current situation. In any event, connections to that IP > address work. The maclist file is definitely complete as far as I can see.Please disregard my comment about 192.168.0.23 in the last email; it was not in your initial email. But that raises the question of what problem you are reporting -- if the log messages aren''t relevant then what is the issue? -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sat, 08 Sep 2012 17:45:34 -0700 Tom Eastep <teastep@shorewall.net> wrote:> On 9/8/12 5:36 PM, Tom Eastep wrote: > > On 9/8/12 5:25 PM, Steve Thompson wrote: > >> On Sat, 8 Sep 2012, Tom Eastep wrote: > >> > >>> Here''s a clue. We see this log message: > >>> > >>> Sep 8 19:14:37 br2_rec:REJECT:IN=br2 OUT=br1 SRC=192.168.0.23 > >>> DST=192.168.4.2 LEN=76 TOS=0x00 PREC=0x00 TTL=127 ID=6379 > >>> PROTO=UDP SPT=123 DPT=123 LEN=56 > >>> > >>> It is being rejected because there is maclist no entry for > >>> 192.168.0.3 on br2 (which is the bridge on which this packet was > >>> received). > >>> > >>> In fact, there is no maclist entry for that IP address at all. > >>> When you see packets being rejected in one of the _rec chains, > >>> you must check your maclist entries. > >> > >> Ah, but there _is_ an entry in the maclist file for this IP > >> address. An extract: > >> > >> ACCEPT br2 84:2B:2B:47:D6:85 192.168.0.3 > >> ACCEPT br2 84:2B:2B:47:D6:86 192.168.0.3 > >> ACCEPT br2 00:1B:21:6F:2B:54 192.168.0.3 > >> ACCEPT br2 00:1B:21:6F:2B:55 192.168.0.3 > >> > > > > 192.168.0.23 IS NOT 192.168.0.3. > > > > And I apologize for yelling :-) But these problems are all about > details. > > -TomYou were still spot on, Tom.> 192.168.0.23 is a virtual IP address that is > implemented by the keepalived daemon on one of the shorewall system''s > virtual machines. It, as such, doesn''t have a MAC address, and isn''t > involved in the current situation. In any event, connections to that > IP address work. The maclist file is definitely complete as far as I > can see.The packets from 192.168.0.23 come in from br2. I had a look at the br2_rec chain. All the IP addresses in there are set to RETURN and the final rule is reject. 192.168.0.23 is not in there. And it seems that arp disagrees with you on it having a MAC address (and it is impossible that an IP has no mac isn''t it? Layer 3 requires layer 2.) ? (192.168.0.23) at 52:54:00:5c:e0:54 [ether] on br2 This mac is also being used by 192.168.0.14, .15 and .17. And iptables is filtering *the IP* which is not explicitly allowed. So it looks like you need to add this to the maclist Steve, I think that will fix it. Out of curiosity, why did you remove ''bridge'' from the OPTIONS in /etc/interfaces? (Tom, sorry for sending this email to you twice, previously I used the wrong From address. Mailman should learn to handle address tags per RFC5233 <mumble>) -- Stay in touch, Mark van Dijk. ,--------------------------------- ----------------------------'' Sun Sep 09 08:27 UTC 2012 Today is Boomtime, the 33rd day of Bureaucracy in the YOLD 3178 ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sun, 9 Sep 2012 10:30:37 +0200 Mark van Dijk <lists+shorewall@internecto.net> wrote:> And it seems that arp disagrees with you on it having a MAC addressErr... on it NOT having an IP address. Critical word. :) ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sun, 9 Sep 2012, Mark van Dijk wrote:> So it looks like you need to add this to the maclist Steve, I think > that will fix it.Please forget about 192.168.0.23; it has nothing to do with the current problem (about which I will post more shortly). It is a virtual IP address in any event and doesn''t have a MAC address. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sun, 9 Sep 2012, Mark van Dijk wrote:> Out of curiosity, why did you remove ''bridge'' from the OPTIONS > in /etc/interfaces?Sorry, missed this the first time around. I did NOT remove the bridge option from the interfaces file. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sat, 8 Sep 2012, Tom Eastep wrote:> But that raises the question of what problem you are reporting -- if the > log messages aren''t relevant then what is the issue?Here is the situation. The shorewall system has an IP address in 192.168.0.0/22 subnet (zone "pub") on bridge br2 (192.168.0.3 and also 192.168.0.1), and an IP address in 192.168.4.0/22 (zone "hpc") on bridge br1 (192.168.4.3 and also 192.168.4.1). The .1 addresses are the default gateways for the clients in the relevant subnet. These addresses can fail over to a second identical shorewall box, but this is not relevant here. The KVM virtual machine on the shorewall box (also CentOS 6.3) has interface eth0 with IP address 192.168.3.254 bridged to br2, and an interface eth1 with IP address 192.168.7.254 bridged to br1. Subnet mask is 255.255.252.0 (/22).>From a client in the 192.168.0.0/22 subnet (IP address 192.168.0.172), Ican issue: ssh 192.168.3.254 and this will successfully log on to the virtual machine, since it is on the same subnet and thus goes directly. I can issue: ssh 192.168.7.254 and this will eventually time out. There are NO log message on the shorewall box; it appears that no packets are passed to the virtual machine (and a traceroute ends at the shorewall box).>From a client in the 192.168.4.0/22 subnet (IP address 192.168.5.241), Ican issue: ssh 192.168.7.254 and this will successfully log on to the virtual machine. I can issue: ssh 192.168.3.254 and this will eventually time out. There are NO log message on the shorewall box; it appears that no packets are passed to the virtual machine (and a traceroute ends at the shorewall box). I have the following entries, amongst others, in /etc/shorewall/policy: pub hpc ACCEPT hpc pub ACCEPT net all DROP info <- internet zone all all REJECT info and in /etc/shorewall/interfaces: hpc br1 - bridge,routeback,tcpflags,nosmurfs pub br2 - bridge,routeback,tcpflags,nosmurfs from which I have removed the maclist option. This is the configuration from which the attached shorewall dump was taken. Removing the maclist option makes no difference to the outcome. I do not understand why packets that reach the firewall from a client on the opposing subnet do not reach the virtual machine, since the policy file would appear to allow them. Again, there are no log messages. For any other machines that are NOT virtual machines on the shorewall box, everything works OK. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 09/09/2012 07:11 AM, Steve Thompson wrote:> On Sat, 8 Sep 2012, Tom Eastep wrote: > >> But that raises the question of what problem you are reporting -- if the >> log messages aren''t relevant then what is the issue? > > Here is the situation. The shorewall system has an IP address in > 192.168.0.0/22 subnet (zone "pub") on bridge br2 (192.168.0.3 and also > 192.168.0.1), and an IP address in 192.168.4.0/22 (zone "hpc") on bridge > br1 (192.168.4.3 and also 192.168.4.1). The .1 addresses are the default > gateways for the clients in the relevant subnet. These addresses can > fail over to a second identical shorewall box, but this is not relevant > here. > > The KVM virtual machine on the shorewall box (also CentOS 6.3) has > interface eth0 with IP address 192.168.3.254 bridged to br2, and an > interface eth1 with IP address 192.168.7.254 bridged to br1. Subnet mask > is 255.255.252.0 (/22). > >> From a client in the 192.168.0.0/22 subnet (IP address 192.168.0.172), I > can issue: > > ssh 192.168.3.254 > > and this will successfully log on to the virtual machine, since it is on > the same subnet and thus goes directly. I can issue: > > ssh 192.168.7.254 > > and this will eventually time out. There are NO log message on the > shorewall box; it appears that no packets are passed to the virtual > machine (and a traceroute ends at the shorewall box). > >> From a client in the 192.168.4.0/22 subnet (IP address 192.168.5.241), I > can issue: > > ssh 192.168.7.254 > > and this will successfully log on to the virtual machine. I can issue: > > ssh 192.168.3.254 > > and this will eventually time out. There are NO log message on the > shorewall box; it appears that no packets are passed to the virtual > machine (and a traceroute ends at the shorewall box). > > I have the following entries, amongst others, in /etc/shorewall/policy: > > pub hpc ACCEPT > hpc pub ACCEPT > net all DROP info <- internet zone > all all REJECT info > > and in /etc/shorewall/interfaces: > > hpc br1 - bridge,routeback,tcpflags,nosmurfs > pub br2 - bridge,routeback,tcpflags,nosmurfs > > from which I have removed the maclist option. This is the configuration > from which the attached shorewall dump was taken. Removing the maclist > option makes no difference to the outcome. > > I do not understand why packets that reach the firewall from a client on > the opposing subnet do not reach the virtual machine, since the policy > file would appear to allow them. Again, there are no log messages. For > any other machines that are NOT virtual machines on the shorewall box, > everything works OK.And if you temporarily ''shorewall clear'', this all works perfectly? -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sun, 9 Sep 2012, Tom Eastep wrote:> And if you temporarily ''shorewall clear'', this all works perfectly?I don''t think it safe for me to try that on a production firewall; I would have to establish a separate test setup. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sun, 9 Sep 2012, Tom Eastep wrote:> And if you temporarily ''shorewall clear'', this all works perfectly?Well, whaddya know. I set up a similar test configuration, which also fails, but it still fails after a ''shorewall clear''. So it is probably something funky about my bridge setup and is not related to shorewall. I can see from tcpdump that packets arrive at the shorewall system but do not make it as far as the virtual machine, being dropped or filtered somewhere. Why, I still don''t know, but at least I have an avenue to investigate. So I thank all of you for your input, and please accept my apologies for wasting your time. Steve ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On 9/9/12 2:54 PM, Steve Thompson wrote:> On Sun, 9 Sep 2012, Tom Eastep wrote: > >> And if you temporarily ''shorewall clear'', this all works perfectly? > > Well, whaddya know. I set up a similar test configuration, which also > fails, but it still fails after a ''shorewall clear''. So it is probably > something funky about my bridge setup and is not related to shorewall. I > can see from tcpdump that packets arrive at the shorewall system but do > not make it as far as the virtual machine, being dropped or filtered > somewhere. Why, I still don''t know, but at least I have an avenue to > investigate. > > So I thank all of you for your input, and please accept my apologies for > wasting your time.No problem. Hope you get it sorted. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today''s security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
On Sun, 9 Sep 2012, Tom Eastep wrote:> On 9/9/12 2:54 PM, Steve Thompson wrote: >> On Sun, 9 Sep 2012, Tom Eastep wrote: >> >>> And if you temporarily ''shorewall clear'', this all works perfectly? >> >> Well, whaddya know. I set up a similar test configuration, which also >> fails, but it still fails after a ''shorewall clear''. So it is probably >> something funky about my bridge setup and is not related to shorewall. I >> can see from tcpdump that packets arrive at the shorewall system but do >> not make it as far as the virtual machine, being dropped or filtered >> somewhere. >> So I thank all of you for your input, and please accept my apologies for >> wasting your time. > > No problem. Hope you get it sorted.The solution to this turned out to be related to the change in the function of the net.ipv4.conf.default.rp_filter parameter in the CentOS 6 kernels; it had nothing to do with KVM. Changing the value of rp_filter from 1 to 2 in KVM1, etc resolved all issues. It was nothing to do with the bridge or shorewall setup. Steve ------------------------------------------------------------------------------ How fast is your code? 3 out of 4 devs don\\\''t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html
On 09/15/2012 05:52 AM, Steve Thompson wrote:> The solution to this turned out to be related to the change in the > function of the net.ipv4.conf.default.rp_filter parameter in the CentOS 6 > kernels; it had nothing to do with KVM. Changing the value of rp_filter > from 1 to 2 in KVM1, etc resolved all issues. It was nothing to do with > the bridge or shorewall setup.Thanks for the update, Steve. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ How fast is your code? 3 out of 4 devs don\\\''t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html