Hi, I''ve configured a couple of virtual machines using libvirt with kvm-qemu. They''re using tun devices (routed, not bridged). Now I want to access some ports of these VMs from the outer world. I set up DNAT rules, restarted shorewall and everything went fine (I even have rules to access the VMs ssh servers from non-standard ports on the host public interface). After having everything set-up I rebooted and couldn''t access the VMs from outside anymore... Checking "iptables -L -v" output I could see the DNAT rules being hit, but the corresponding ACCEPT rules were never hit. After a LOT of guessing and trying without any success, I simply restarted shorewall and everything was nice again... Now I think (I''m pretty sure, actually) that the problem is that shorewall is starting earlier than libvirt-bin and I suppose that since the tun devices used by qemu don''t exist yet, then the rules are not generated correctly. I don''t think I want to delay shorewall''s startup... I want it up ASAP when I''m booting... but it seems I''d need to restart it after starting libvirt-bin... What would be the "cleanest" way to do this? I''m using ubuntu 8.10 server with stock libvirt & kvm packages. I''m using shorewall-perl 4.2.5. TIA -- Mariano Absatz - "El Baby" el.baby@gmail.com www.clueless.com.ar -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Friends help you move. Real friends help you move bodies. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- * TagZilla 0.066 * http://tagzilla.mozdev.org ------------------------------------------------------------------------------
Mariano Absatz wrote:> After a LOT of guessing and trying without any success, I simply > restarted shorewall and everything was nice again... > > Now I think (I''m pretty sure, actually) that the problem is that > shorewall is starting earlier than libvirt-bin and I suppose that since > the tun devices used by qemu don''t exist yet, then the rules are not > generated correctly. > > I don''t think I want to delay shorewall''s startup... I want it up ASAP > when I''m booting... but it seems I''d need to restart it after starting > libvirt-bin...I personally would fix my Shorewall configuration so it didn''t require the restart. Unfortunately, your post gives us no information that would let us help you with that. Please see http://www.shorewall.net/support.htm#Guidelines. -Tom PS -- If you still want to restart Shorewall when Libvirt starts, you''ll have to ask the Libvirt folks how to do that. -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------
Tom Eastep escribió el 30/03/09 11:21:> Mariano Absatz wrote: > >> After a LOT of guessing and trying without any success, I simply >> restarted shorewall and everything was nice again... >> >> Now I think (I''m pretty sure, actually) that the problem is that >> shorewall is starting earlier than libvirt-bin and I suppose that since >> the tun devices used by qemu don''t exist yet, then the rules are not >> generated correctly. >> >> I don''t think I want to delay shorewall''s startup... I want it up ASAP >> when I''m booting... but it seems I''d need to restart it after starting >> libvirt-bin... >> > > I personally would fix my Shorewall configuration so it didn''t require > the restart. Unfortunately, your post gives us no information that would > let us help you with that. Please see > http://www.shorewall.net/support.htm#Guidelines. >Hi Tom, thanx for your reply. Sorry for not having included all the required information. I now left 4 different dumps in http://ybab.net/shorewall/ Here are the commands and their output: $ sudo /sbin/shorewall version 4.2.5 $ sudo ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1e:c9:b0:70:e2 brd ff:ff:ff:ff:ff:ff inet 94.75.244.29/26 brd 94.75.244.63 scope global eth0 inet 94.75.244.57/26 brd 94.75.244.63 scope global secondary eth0:0 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 00:1e:c9:b0:70:e4 brd ff:ff:ff:ff:ff:ff 4: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 26:1f:34:97:99:a4 brd ff:ff:ff:ff:ff:ff inet 10.3.14.1/24 brd 10.3.14.255 scope global vnet0 5: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether 32:40:38:8b:46:0b brd ff:ff:ff:ff:ff:ff 6: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether 26:1f:34:97:99:a4 brd ff:ff:ff:ff:ff:ff $ sudo ip route show 94.75.244.0/26 dev eth0 proto kernel scope link src 94.75.244.29 10.3.14.0/24 dev vnet0 proto kernel scope link src 10.3.14.1 default via 94.75.244.62 dev eth0 metric 100 now, on to the problem description. I bootstrap the server. It boots OK, shorewall ran and the rules are in place. I execute: $ sudo /sbin/shorewall dump > dump-00.txt I try to connect to the ssh server in the virtual machines from outside the server using the following: $ ssh -p 10017 cejil@94.75.244.57 The connection hangs and eventually times out. While waiting for it to timeout, in the server, I execute: $ sudo /sbin/shorewall dump > dump-01.txt Then, I issue $ sudo invoke-rc.d shorewall restart to restart shorewall. I execute: $ sudo /sbin/shorewall dump > dump-10.txt I try to connect again from outside: $ ssh -p 10017 cejil@94.75.244.57 The connection succeeds, and once again, I execute: $ sudo /sbin/shorewall dump > dump-11.txt Note that, whenever I can''t connect to the virtual machine from outside, if I''m logged into the host and issue ssh cejil@10.3.14.17 it does work OK, that is, the virtual machine is up and running and in the expected address. Let me know if there''s something else you''d like me to do, and thanx again. -- Mariano Absatz - "El Baby" el.baby@gmail.com www.clueless.com.ar -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Unix is very simple, but it takes a genius to understand the simplicity. -- Dennis Ritchie -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- * TagZilla 0.066 * http://tagzilla.mozdev.org ------------------------------------------------------------------------------
Mariano Absatz wrote:> The connection hangs and eventually times out. While waiting for it to > timeout, in the server, I execute: > $ sudo /sbin/shorewall dump > dump-01.txt >The Netfilter ruleset at this point is clearly wrong. Please: a) sudo /sbin/shorewall show -f capabilities > /etc/shorewall/caps b) sudo tar -zcf shorewall.tgz /etc/shorewall c) Send shorewall.tgz as an attachment to upload@shorewall.net Thanks! -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------
Tom Eastep wrote:> Mariano Absatz wrote: > >> The connection hangs and eventually times out. While waiting for it to >> timeout, in the server, I execute: >> $ sudo /sbin/shorewall dump > dump-01.txt >> > > The Netfilter ruleset at this point is clearly wrong. Please: > > a) sudo /sbin/shorewall show -f capabilities > /etc/shorewall/caps > b) sudo tar -zcf shorewall.tgz /etc/shorewall > c) Send shorewall.tgz as an attachment to upload@shorewall.net >Looking at this some more, I believe that when libvirt starts, it is inserting rules into the FORWARD chain. I''m quite certain that the extra rules I''m seeing in the first two dumps are not being created by Shorewall. I wish that these virtualization products would keep their hands off of the Netfilter configuration.... I suggest that you check the libvirt documentation to see if there isn''t a way to stop it from inserting these rules. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------
Tom Eastep escribió el 31/03/09 11:40:> Tom Eastep wrote: > >> Mariano Absatz wrote: >> >> >>> The connection hangs and eventually times out. While waiting for it to >>> timeout, in the server, I execute: >>> $ sudo /sbin/shorewall dump > dump-01.txt >>> >>> >> The Netfilter ruleset at this point is clearly wrong. Please: >> >> a) sudo /sbin/shorewall show -f capabilities > /etc/shorewall/caps >> b) sudo tar -zcf shorewall.tgz /etc/shorewall >> c) Send shorewall.tgz as an attachment to upload@shorewall.net >> >> > > Looking at this some more, I believe that when libvirt starts, it is > inserting rules into the FORWARD chain. I''m quite certain that the extra > rules I''m seeing in the first two dumps are not being created by > Shorewall. I wish that these virtualization products would keep their > hands off of the Netfilter configuration.... > > I suggest that you check the libvirt documentation to see if there isn''t > a way to stop it from inserting these rules. >Boy are you right!!! libvirt is so adding iptables rules... it does have some merit... it tries to be an easy interface to adding virtual machines and it tries to solve networking problems easily even when you don''t have a clue about routing, nating or firewalling. I already defined a new isolated network in libvirt but can''t convince it not to add some rules... I''ll post to libvirt list, if I come with a reasonable answer, I''ll try to summarize it in the wiki. Thanx again for all your help, Tom. -- Mariano Absatz - "El Baby" el.baby@gmail.com www.clueless.com.ar -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- All syllogisms contain three lines. Therefore this is not a syllogism. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- * TagZilla 0.066 * http://tagzilla.mozdev.org ------------------------------------------------------------------------------
Mariano Absatz
2009-Apr-02 16:00 UTC
virtual machine 2 virtual machine rules (was Re: restarting shorewall after libvirt-bin starts)
Hi again, I''m still dealing with my libvirt issues but it is now clear that it''s not a shorewall problem. Now I have a related problem... since my host is the router for all virtual machines, it has to route trafic among them. The problem is that all the trafic, from the host point of view, is seen on the same interface (in my case, vnet0). The point is that I have configured the following: ####################### interfaces ####################### net eth0 - tcpflags,logmartians,nosmurfs,norfc1918,blacklist vms vnet0 - bridge,tcpflags,nosmurfs,blacklist vpn tun+ - tcpflags,nosmurfs,blacklist and a few rules to allow for "intra-vms" trafic like these: ####################### rules ####################### ACCEPT vms vms icmp SSH/ACCEPT vms vms DNS/ACCEPT vms vms However, these rules are never invoked. That is, the vms2vms chain is created but not referred to. Is there any way to convince shorewall to refer to these rules? I see "BRIDGING=Yes" is not supported in shorewall-perl... The output of ''shorewall dump'' is at http://ybab.net/shorewall/status-20090402a.txt I just did: sudo sh -c "/sbin/shorewall show -f capabilities > /etc/shorewall/caps" sudo tar -zcf shorewall.tgz /etc/shorewall If you want, I can send it to upload@shorewall.net Regards. -- Mariano Absatz - "El Baby" el.baby@gmail.com www.clueless.com.ar ------------------------------------------------------------------------------
Tom Eastep
2009-Apr-02 16:16 UTC
Re: virtual machine 2 virtual machine rules (was Re: restarting shorewall after libvirt-bin starts)
Mariano Absatz wrote:> Hi again, > > I''m still dealing with my libvirt issues but it is now clear that it''s > not a shorewall problem. > > Now I have a related problem... since my host is the router for all > virtual machines, it has to route trafic among them. The problem is that > all the trafic, from the host point of view, is seen on the same > interface (in my case, vnet0). > > The point is that I have configured the following: > > ####################### interfaces ####################### > net eth0 - > tcpflags,logmartians,nosmurfs,norfc1918,blacklist > vms vnet0 - bridge,tcpflags,nosmurfs,blacklist > vpn tun+ - tcpflags,nosmurfs,blacklist > > > and a few rules to allow for "intra-vms" trafic like these: > > ####################### rules ####################### > ACCEPT vms vms icmp > SSH/ACCEPT vms vms > DNS/ACCEPT vms vms > > > However, these rules are never invoked. > > That is, the vms2vms chain is created but not referred to. > > Is there any way to convince shorewall to refer to these rules? > > I see "BRIDGING=Yes" is not supported in shorewall-perl...Please read http://www.shorewall.net/bridge-Shorewall-perl.html -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------
Tom Eastep
2009-Apr-02 16:22 UTC
Re: virtual machine 2 virtual machine rules (was Re: restarting shorewall after libvirt-bin starts)
Mariano Absatz wrote:> Hi again, > > I''m still dealing with my libvirt issues but it is now clear that it''s > not a shorewall problem. > > Now I have a related problem... since my host is the router for all > virtual machines, it has to route trafic among them.No it does not. All of the virtual machines are connected to a bridge -- there is no IPv4 routing involved in traffic between virtual machines. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------
Tom Eastep
2009-Apr-02 17:17 UTC
Re: virtual machine 2 virtual machine rules (was Re: restarting shorewall after libvirt-bin starts)
Tom Eastep wrote:> Mariano Absatz wrote: >> Hi again, >> >> I''m still dealing with my libvirt issues but it is now clear that it''s >> not a shorewall problem. >> >> Now I have a related problem... since my host is the router for all >> virtual machines, it has to route trafic among them. The problem is that >> all the trafic, from the host point of view, is seen on the same >> interface (in my case, vnet0). >> >> The point is that I have configured the following: >> >> ####################### interfaces ####################### >> net eth0 - >> tcpflags,logmartians,nosmurfs,norfc1918,blacklist >> vms vnet0 - bridge,tcpflags,nosmurfs,blacklist >> vpn tun+ - tcpflags,nosmurfs,blacklist >> >> >> and a few rules to allow for "intra-vms" trafic like these: >> >> ####################### rules ####################### >> ACCEPT vms vms icmp >> SSH/ACCEPT vms vms >> DNS/ACCEPT vms vms >> >> >> However, these rules are never invoked. >> >> That is, the vms2vms chain is created but not referred to. >> >> Is there any way to convince shorewall to refer to these rules? >> >> I see "BRIDGING=Yes" is not supported in shorewall-perl... > > Please read http://www.shorewall.net/bridge-Shorewall-perl.htmlIf you just want to allow all traffic between the VMs: a) Delete all of those silly rules. b) Remove the ''bridge'' option from vnet0 in /etc/shorewall/interfaces c) Add the ''routeback'' option to vnet0 in /etc/shorewall/interfaces If you only want to allow DNS and SSH: a) Add a vms->vms REJECT policy to /etc/shorewall/interfaces. b) Remove the ''bridge'' option from vnet0 in /etc/shorewall/interfaces c) Add the ''routeback'' option to vnet0 in /etc/shorewall/interfaces If you want finer grained control, you probably want to refer to the article I mention above. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------
Mariano Absatz
2009-Apr-02 20:37 UTC
Re: virtual machine 2 virtual machine rules (was Re: restarting shorewall after libvirt-bin starts)
Tom Eastep escribió el 02/04/09 14:17:> Tom Eastep wrote: > >> Please read http://www.shorewall.net/bridge-Shorewall-perl.html >>Thanx... I skimmed over it and I started 2 understand... anyway it does not refer to my case since I''m not firewalling the bridge using 2 interfaces in my host...> > If you just want to allow all traffic between the VMs: > > a) Delete all of those silly rules. > b) Remove the ''bridge'' option from vnet0 in /etc/shorewall/interfaces > c) Add the ''routeback'' option to vnet0 in /etc/shorewall/interfaces > > If you only want to allow DNS and SSH: > > a) Add a vms->vms REJECT policy to /etc/shorewall/interfaces. > b) Remove the ''bridge'' option from vnet0 in /etc/shorewall/interfaces > c) Add the ''routeback'' option to vnet0 in /etc/shorewall/interfaces >Now, this is (I think) precisely what I needed... In this case I _do_ need the silly rules, don''t I? I already had the REJECT policy, but I have it in ''policies'' rather than ''interfaces''... is there a difference? Now I think I understand what libvirt is doing: It inserts rules at the top of INPUT to allow dns/bootp requests from vnet0 to anywhere Chain INPUT (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT udp -- vnet0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 0 0 ACCEPT tcp -- vnet0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 0 0 ACCEPT udp -- vnet0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 0 0 ACCEPT tcp -- vnet0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67 It inserts rules at the top of FORWARD rules to allow any trafic from vnet0 to vnet0 and blocks all other trafic to/from it (I guess this is because I defined it as an "isolated" network hoping it would insert nothing at all): Chain FORWARD (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination - 0 0 ACCEPT all -- vnet0 vnet0 0.0.0.0/0 0.0.0.0/0 - 0 0 REJECT all -- * vnet0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable - 0 0 REJECT all -- vnet0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable -- Mariano Absatz - "El Baby" el.baby@gmail.com www.clueless.com.ar -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Is sloppiness in speech caused by ignorance or apathy? I don''t know and I don''t care. William Safire US columnist & speechwriter (1929 - ) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- * TagZilla 0.066 * http://tagzilla.mozdev.org ------------------------------------------------------------------------------
Tom Eastep
2009-Apr-02 20:46 UTC
Re: virtual machine 2 virtual machine rules (was Re: restarting shorewall after libvirt-bin starts)
Mariano Absatz wrote:> Tom Eastep escribió el 02/04/09 14:17: >> Tom Eastep wrote: >> >>> Please read http://www.shorewall.net/bridge-Shorewall-perl.html >>> > Thanx... I skimmed over it and I started 2 understand... anyway it does > not refer to my case since I''m not firewalling the bridge using 2 > interfaces in my host... >> >> If you just want to allow all traffic between the VMs: >> >> a) Delete all of those silly rules. >> b) Remove the ''bridge'' option from vnet0 in /etc/shorewall/interfaces >> c) Add the ''routeback'' option to vnet0 in /etc/shorewall/interfaces >> >> If you only want to allow DNS and SSH: >> >> a) Add a vms->vms REJECT policy to /etc/shorewall/interfaces. >> b) Remove the ''bridge'' option from vnet0 in /etc/shorewall/interfaces >> c) Add the ''routeback'' option to vnet0 in /etc/shorewall/interfaces >> > Now, this is (I think) precisely what I needed... > > In this case I _do_ need the silly rules, don''t I?Yes.> > I already had the REJECT policy, but I have it in ''policies'' rather than > ''interfaces''... is there a difference?That was a typo -- I meant /etc/shorewall/policy. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------