Hello!
Since i could not find any information on the internet about this subject,
i'm going to try my luck on this list.
I'm trying to setup network-filter on a routed setup. I have a root-server
at Hetzner, a german hosting provider.
Along with my server i ordered a (/28) subnet to be able to setup dedicated IPs
for my virtual machines (KVM).
My Server is running Ubuntu 12.04 with libvirt 0.9.8 . Since Hetzner does not
allow any bridged traffic, i had
to setup a routed network. Currently my (via libvirt) defined network looks like
this:
(lets assume my subnet is 1.2.3.64/28):
<network>
<name>hetzner-subnet-v4</name>
<forward dev='eth0' mode='route'>
<interface dev='eth0'/>
</forward>
<bridge name='route-br0' stp='off' delay='0' />
<mac address='52:54:00:F0:D0:AA'/>
<ip address='1.2.3.65'
netmask='255.255.255.240'></ip>
</network>
The network definition for all running VMs looks like this:
<interface type='network'>
<mac address='52:54:00:00:00:##'/>
<source network='hetzner-subnet-v4'/>
</interface>
Without using Network-Filters, this setup is running as expected. All traffic is
correctly forwarded to my virtual
machines connected to "route-br0" and the following iptables-rules are
created in the FORWARD Chain:
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
target prot opt in out source destination
ACCEPT all -- eth0 route-br0 0.0.0.0/0 1.2.3.64/28
ACCEPT all -- route-br0 eth0 1.2.3.64/28 0.0.0.0/0
ACCEPT all -- route-br0 route-br0 0.0.0.0/0 0.0.0.0/0
REJECT all -- * route-br0 0.0.0.0/0 0.0.0.0/0
reject-with icmp-port-unreachable
REJECT all -- route-br0 * 0.0.0.0/0 0.0.0.0/0
reject-with icmp-port-unreachable
When i try to setup a network-filter for a VM (a modified version of
http://libvirt.org/formatnwfilter.html last example):
<filter name='server-x' chain='root'>
<filterref filter='clean-traffic'/>
<rule action='accept' direction='in'
priority='500'>
<all state='ESTABLISHED'/>
</rule>
<rule action='accept' direction='out'
priority='500'>
<all state='ESTABLISHED,RELATED'/>
</rule>
<rule action='accept' direction='in'
priority='500'>
<tcp state='NEW' dstportstart='22'/>
</rule>
<rule action='accept' direction='out'
priority='500'>
<all state='NEW'/>
</rule>
<rule action='drop' direction='inout'
priority='500'>
<all/>
</rule>
</filter>
and adding the filter to my interface-definition of a VM using the following
syntax:
<filterref filter='server-x'>
<parameter name='IP' value='1.2.3.70'/>
</filterref>
additional iptable-rules are getting created. The problematic rule seems to be
the following:
-A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX
which should trigger the following rules:
-A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT
-A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT
But this actually never happens. The FO-vnetX Chain never sees any packets and
my syslog says:
xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains
for non-bridged traffic is not supported anymore
Am i doing something wrong? I hope i did not write too much useless stuff here.
I'm tried to figure it all out by
myself, but im currently stuck. Lets hope some wise guys can help me out here.
Maybe there is some documentation i have missed?
Thanks!
kind regards,
Sebastian
On 02/14/2014 08:40 PM, h0rst wrote:> Hello! > > Since i could not find any information on the internet about this subject, i'm going to try my luck on this list. > > I'm trying to setup network-filter on a routed setup. I have a root-server at Hetzner, a german hosting provider. > Along with my server i ordered a (/28) subnet to be able to setup dedicated IPs for my virtual machines (KVM). > My Server is running Ubuntu 12.04 with libvirt 0.9.8 . Since Hetzner does not allow any bridged traffic,You *really* should upgrade to a newer libvirt.> Without using Network-Filters, this setup is running as expected. All traffic is correctly forwarded to my virtual > machines connected to "route-br0" and the following iptables-rules are created in the FORWARD Chain: > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) > target prot opt in out source destination > ACCEPT all -- eth0 route-br0 0.0.0.0/0 1.2.3.64/28 > ACCEPT all -- route-br0 eth0 1.2.3.64/28 0.0.0.0/0 > ACCEPT all -- route-br0 route-br0 0.0.0.0/0 0.0.0.0/0 > REJECT all -- * route-br0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable > REJECT all -- route-br0 * 0.0.0.0/0Those previous two rules are the ones added when you specify a forward dev. You don't need to do that - I would recommend removing the "dev='eth0'" from the <forward> element of the network, along with the "<interface dev='eth0'/>" subelement. This won't change operation at all, it will just make things slightly less confusing and misleading.> 0.0.0.0/0 reject-with icmp-port-unreachable > > When i try to setup a network-filter for a VM (a modified version of http://libvirt.org/formatnwfilter.html last example): > > <filter name='server-x' chain='root'> > <filterref filter='clean-traffic'/> > <rule action='accept' direction='in' priority='500'> > <all state='ESTABLISHED'/> > </rule> > <rule action='accept' direction='out' priority='500'> > <all state='ESTABLISHED,RELATED'/> > </rule> > <rule action='accept' direction='in' priority='500'> > <tcp state='NEW' dstportstart='22'/> > </rule> > <rule action='accept' direction='out' priority='500'> > <all state='NEW'/> > </rule> > <rule action='drop' direction='inout' priority='500'> > <all/> > </rule> > </filter> > > and adding the filter to my interface-definition of a VM using the following syntax: > > <filterref filter='server-x'> > <parameter name='IP' value='1.2.3.70'/> > </filterref> > > additional iptable-rules are getting created. The problematic rule seems to be the following: > > -A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX > > which should trigger the following rules: > > -A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT > -A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT > > But this actually never happens. The FO-vnetX Chain never sees any packets and my syslog says: > > xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymoreThat somehow sounded familiar, so I looked it up in the git history and found this: http://libvirt.org/git/?p=libvirt.git;a=commit;h=65fb9d49cc9caae210977934b53d87e56429407b That patch was included in libvirt-1.0.2, just about a year ago.> > Am i doing something wrong?YOu need to upgrade your libvirt to at least 1.0.2 (preferably newer).
On Di, 2014-02-18 at 12:03 +0200, Laine Stump wrote:> You *really* should upgrade to a newer libvirt.I know that version 0.9.8 is very old. But to be honest i tried to avoid upgrading and compiling a newer version since i don't know if it has any effects on running VMs (but i haven't checked this yet). Its a production server and i did not want to interrupt any services running on these VMs. However, i'm afraid that upgrading might be the only option if i want to avoid setting up iptables manually.> Those previous two rules are the ones added when you specify a forward > dev. You don't need to do that - I would recommend removing the > "dev='eth0'" from the <forward> element of the network, along with the > "<interface dev='eth0'/>" subelement. This won't change operation at > all, it will just make things slightly less confusing and misleading.Thank you for pointing this out. I just did that. And after a reboot everything is still working as expected (yes. I just DID interrupt the services running in my VMs. So i guess i could even upgrade to a newer libvirt =) )> > > > -A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX > > > > which should trigger the following rules: > > > > -A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT > > -A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT > > > > But this actually never happens. The FO-vnetX Chain never sees any packets and my syslog says: > > > > xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore> That somehow sounded familiar, so I looked it up in the git history and > found this: > > http://libvirt.org/git/?p=libvirt.git;a=commit;h=65fb9d49cc9caae210977934b53d87e56429407b > > That patch was included in libvirt-1.0.2, just about a year ago.After reading that i remove the following iptables rule: iptables -D libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX and manually added this rule: (the patch said that adding an extra argument (--physdev-is-bridged) is needed for rules like this): iptables -A libvirt-out -m physdev --physdev-is-bridged --physdev-out vnetX -g FO-vnetX Indeed this prevents my syslog from being spammed with the mentioned warning. However, this did not fix the problem. This rule never matches anything, and thus the FO-vnetX Chain never sees any packets. Using a testing rule like: iptables -A libvirt-out -d 1.2.3.70 -g FO-vnetX made everything work as expected. Well, this is definitely not the way it is expected to work since it does not do any "bridge port" matching. This all makes me think its not a libvirt specific problem and updating to a newer version will not fix my problem. Maybe there is somebody out there using a setup like mine and can show me the rules that a getting created with a newer version of libvirt? So long, thank you for all the usefull information! Kind regards, Sebastian