Hi Everyone, Does anyone know any rules that I could use (using iptable, ebtables, or otherwise) that could force all traffic coming from a guest to go out via a particular interface? I wish to stop "inter-guest" communication, without going via my firewall first. Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 13/06/10 16:52, 0bo0 wrote:> hi, > > On Sun, Jun 13, 2010 at 8:45 AM, Jonathan Tripathy<jonnyt@abpni.co.uk> wrote: > >> Does anyone know any rules that I could use (using iptable, ebtables, or >> otherwise) that could force all traffic coming from a guest to go out via a >> particular interface? I wish to stop "inter-guest" communication, without >> going via my firewall first. >> > tho not sure it addresses your specific issue, you may find this of > general interest when deploying a fw in a Xen VM: > > http://www.shorewall.net/XenMyWay.html > > hth >Thanks for that. I''ve already read that before, but that post talks about ip filtering using iptables+shorewall. I wish to stop frames at the ethernet layer by forcing all traffic out via a paticular interface. Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Jun 13, 2010 at 10:45 PM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:> Hi Everyone, > > Does anyone know any rules that I could use (using iptable, ebtables, or > otherwise) that could force all traffic coming from a guest to go out via a > particular interface? I wish to stop "inter-guest" communication, without > going via my firewall first.IIRC Xen bridged networking by default passes domU traffic through the bridge on dom0 (even for inter-guest communications). Try setting up some rules there (i.e. make dom0 your firewall). If you want to use an external firewall (not in dom0), then no, I don''t know of any way to do that. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 13/06/10 17:02, Fajar A. Nugraha wrote:> On Sun, Jun 13, 2010 at 10:45 PM, Jonathan Tripathy<jonnyt@abpni.co.uk> wrote: > >> Hi Everyone, >> >> Does anyone know any rules that I could use (using iptable, ebtables, or >> otherwise) that could force all traffic coming from a guest to go out via a >> particular interface? I wish to stop "inter-guest" communication, without >> going via my firewall first. >> > IIRC Xen bridged networking by default passes domU traffic through the > bridge on dom0 (even for inter-guest communications). Try setting up > some rules there (i.e. make dom0 your firewall). > > If you want to use an external firewall (not in dom0), then no, I > don''t know of any way to do that. > >Hi Fajar, I''m pretty sure that by using ebtables (in the Dom0) at the "link layer", I can force all traffic out via one interface. I believe that a "linux-bridge" acts just like a stupid "Hub" (Ah remember those days before switches were common?!). So by only allowing traffic out via the interface which is connected to the firewall, traffic if forced to go out that way. Of course, this is just theory, so I''m asking here as someone else may have some experience. My backup plan, as you rightfully mention, is to just do the firewalling in the Dom0 itself. I''d just like to use a single external firewall for easy management. Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Jonathan, I read your mail and those you posted in different previous threads and I think that you should probably consider *not* using a bridge and using pure routing instead: - Do you really need bridge-only features (especially broadcasts from domU to domU or broadcasts trespassing the dom0)? If I understand your plans correctly, you want all your domUs to be isolated with their own IP address and only communicating via a dedicated firewall. This way, you would not need broadcasts between clients (this is only interesting if you want to use LAN services between your domUs, because broadcasts are not sent across the internet anyway). - AFAIK, routing is more secure and faster than bridging, but somewhat harder to setup. - You could do what you posted below with routing. It might work with bridging, too, but I don''t know a good way to do that with a bridge. With routing, you would need policy routing because of this elementary problem: You have (to make things easier to explain, in this example only 2) two DomUs (let''s say, 1.0.0.1 on vif-1.0 and 1.0.0.2 on vif-2.0), the Domain-0 and a dedicated firewall between the Dom0 and the internet. If 1.0.0.1 wants to reach any server on the internet (or vice versa), it will trespass the firewall by default. But if 1.0.0.1 wants to send (e.g. an e-mail) to 1.0.0.2 or (more dangerous) wants to attack 1.0.0.2, they would only communicate via the Domain-0 (without the firewall). The problem is: If you route 1.0.0.2 to vif-2.0 under all circumstances, it will bypass the firewall if 1.0.0.1 sent the package. If you route everything except 1.0.0.1 via eth0, you wont be able to reach 1.0.0.2 any way. The solution is: You need to do policy routing. If a package originates from the internet an should be sent to 1.0.0.2, it must be routed to vif-2.0. But if it originates from 1.0.0.1, it must be routed to eth0, so that it is sent to the firewall. The firewall will then process the package and return it to the server, which now must route the package to vif-2.0. So it will take two policy routes: route 1.0.0.2 via vif-2.0, if it is from eth0 route 1.0.0.2 via eth0, if it is not from eth0 I don''t think that those routes would work with a bridge, so consider using routing. Felix Am 13.06.2010 17:45, schrieb Jonathan Tripathy:> Hi Everyone, > > Does anyone know any rules that I could use (using iptable, ebtables, > or otherwise) that could force all traffic coming from a guest to go > out via a particular interface? I wish to stop "inter-guest" > communication, without going via my firewall first. > > Thanks > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Felix, Excellent plan! We are getting closer! I should really put all these wonderful tips from everyone here on a blog or something.. Back to the plan.. The only difficulty I see here is that the DomUs will be using public IP address, and the firewall (between the Internet and Dom0) will be a "filtering bridge" in its own right. However maybe that doesn''t matter. Would you maybe be able to give me some example of the actual rules that I could use? This would be very much appreciated, and if I saw the rules I could work out if my firewall setup is a problem. It would be nice if my ISP just gave my firewall''s WAN interface a single address, and allowed multiple public subnets to be routed via my firewall (so my firewall would act like a router, not a bridge), however I don''t think this is the case. I think all that I will get is just an ethernet cable connected to a switch.. Thanks On 13/06/10 17:10, Felix Kuperjans wrote:> Hi Jonathan, > > I read your mail and those you posted in different previous threads and > I think that you should probably consider *not* using a bridge and using > pure routing instead: > - Do you really need bridge-only features (especially broadcasts from > domU to domU or broadcasts trespassing the dom0)? If I understand your > plans correctly, you want all your domUs to be isolated with their own > IP address and only communicating via a dedicated firewall. This way, > you would not need broadcasts between clients (this is only interesting > if you want to use LAN services between your domUs, because broadcasts > are not sent across the internet anyway). > - AFAIK, routing is more secure and faster than bridging, but somewhat > harder to setup. > - You could do what you posted below with routing. It might work with > bridging, too, but I don''t know a good way to do that with a bridge. > > With routing, you would need policy routing because of this elementary > problem: > > You have (to make things easier to explain, in this example only 2) two > DomUs (let''s say, 1.0.0.1 on vif-1.0 and 1.0.0.2 on vif-2.0), the > Domain-0 and a dedicated firewall between the Dom0 and the internet. > If 1.0.0.1 wants to reach any server on the internet (or vice versa), it > will trespass the firewall by default. > But if 1.0.0.1 wants to send (e.g. an e-mail) to 1.0.0.2 or (more > dangerous) wants to attack 1.0.0.2, they would only communicate via the > Domain-0 (without the firewall). > > The problem is: > > If you route 1.0.0.2 to vif-2.0 under all circumstances, it will bypass > the firewall if 1.0.0.1 sent the package. > If you route everything except 1.0.0.1 via eth0, you wont be able to > reach 1.0.0.2 any way. > > The solution is: > > You need to do policy routing. > If a package originates from the internet an should be sent to 1.0.0.2, > it must be routed to vif-2.0. But if it originates from 1.0.0.1, it must > be routed to eth0, so that it is sent to the firewall. > The firewall will then process the package and return it to the server, > which now must route the package to vif-2.0. > > So it will take two policy routes: > route 1.0.0.2 via vif-2.0, if it is from eth0 > route 1.0.0.2 via eth0, if it is not from eth0 > > I don''t think that those routes would work with a bridge, so consider > using routing. > > Felix > > Am 13.06.2010 17:45, schrieb Jonathan Tripathy: > >> Hi Everyone, >> >> Does anyone know any rules that I could use (using iptable, ebtables, >> or otherwise) that could force all traffic coming from a guest to go >> out via a particular interface? I wish to stop "inter-guest" >> communication, without going via my firewall first. >> >> Thanks >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >> > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Jun 13, 2010 at 11:06 PM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:> I''m pretty sure that by using ebtables (in the Dom0) at the "link layer", I > can force all traffic out via one interface. I believe that a "linux-bridge" > acts just like a stupid "Hub" (Ah remember those days before switches were > common?!).Do you know of any way to "force" all traffic via one interface on a "stupid hub" or a switch?> So by only allowing traffic out via the interface which is > connected to the firewall, traffic if forced to go out that way."allowing" is somewhat easier. Try "man iptables", look for "--in-interface" and "--out-interface", and create the rules on forward chain. So in a sense, "blocking all inter domU traffic" is easy enough. However, if you want to "redirect all inter domU traffice to an external firewall", then it''s won''t be easy (if at all possible).> > Of course, this is just theory, so I''m asking here as someone else may have > some experience. > > My backup plan, as you rightfully mention, is to just do the firewalling in > the Dom0 itself. I''d just like to use a single external firewall for easy > management.Do you have a working solution to prevent physical hosts on the same network from communicating with each other directly? If yes, adapting it to Xen is easy, since in bridged mode Xen behaves similar to a switch. Some colo provider does this by allocating small subnets (/30 or /29) on different vlans for each customer''s server, effectively forcing them to route all traffic via provider''s router (at the cost of many wasted IP addresses). This approach can be adapted easily using many bridges (one for each vlan) on Xen. Another approach is the one used by some DSL providers, forcing their users to use PPPoE. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 13/06/10 17:24, Fajar A. Nugraha wrote:> On Sun, Jun 13, 2010 at 11:06 PM, Jonathan Tripathy<jonnyt@abpni.co.uk> wrote: > >> I''m pretty sure that by using ebtables (in the Dom0) at the "link layer", I >> can force all traffic out via one interface. I believe that a "linux-bridge" >> acts just like a stupid "Hub" (Ah remember those days before switches were >> common?!). >> > Do you know of any way to "force" all traffic via one interface on a > "stupid hub" or a switch? >Well, using ebtables (or something else), I was hoping to turn my "stupid hub" into a "plumbed" hub. By "stupid hub", I mean a device that doesn''t map MAC addresses to ports, which I don''t believe a linux bridge will do. I''m pretty sure some Cisco and HP switches can do this, but that''s irrelevant as we''re dealing with the bridge inside the Dom0.> >> So by only allowing traffic out via the interface which is >> connected to the firewall, traffic if forced to go out that way. >> > "allowing" is somewhat easier. Try "man iptables", look for > "--in-interface" and "--out-interface", and create the rules on > forward chain. > So in a sense, "blocking all inter domU traffic" is easy enough. > However, if you want to "redirect all inter domU traffice to an > external firewall", then it''s won''t be easy (if at all possible). >This is why I wanted to use bridging, instead of IP routing, as since I believe that all ports on a bridge in linux are "promiscuous", by just blocking traffic between DomUs at the "link-layer" may do the trick. Again, only just a theory..> >> Of course, this is just theory, so I''m asking here as someone else may have >> some experience. >> >> My backup plan, as you rightfully mention, is to just do the firewalling in >> the Dom0 itself. I''d just like to use a single external firewall for easy >> management. >> > Do you have a working solution to prevent physical hosts on the same > network from communicating with each other directly? If yes, adapting > it to Xen is easy, since in bridged mode Xen behaves similar to a > switch. Some colo provider does this by allocating small subnets (/30 > or /29) on different vlans for each customer''s server, effectively > forcing them to route all traffic via provider''s router (at the cost > of many wasted IP addresses). This approach can be adapted easily > using many bridges (one for each vlan) on Xen. Another approach is the > one used by some DSL providers, forcing their users to use PPPoE. > >No working solution yet. Hopefully soon Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Jonathan, I think it would be better to do that in two steps: Firstly, get rid of your bridge and get a running configuration with routing. Secondly, you can try to add those policy routing rules, to enforce what you really want. But policy routing is a highly complex and advanced technique. I''ll explain the first step to you know, but I need to setup the second step myself, I can''t do that spontaneously. By the way, you''ll need policy routing built into your dom0 kernel or at least as a module (it''s under advanced routing in the configuration) and iproute2 (should be pulled in as one of Xen''s dependencies). To change to routing, you only need to use another network and vif-script, but I usually write an own one (you''ll need that for those policy routing rules anyway). Important is, that IP forwarding (and if you like, IPv6 forwarding) is enabled. Do that in your network script: # Enbale IP forwarding: echo 1 >/proc/sys/net/ipv4/ip_forward # Enable forwarding for all devices: for dev in $(ls /proc/sys/net/ipv4/conf/) ; do echo 1 >"/proc/sys/net/ipv4/conf/${dev}/forwarding" echo 1 >"/proc/sys/net/ipv6/conf/${dev}/forwarding" done # Disable forwarding for lo: echo 0 >/proc/sys/net/ipv4/conf/lo/forwarding echo 0 >/proc/sys/net/ipv6/conf/lo/forwarding Note, that there are multiple ways to handle routing + ARP: - The default Xen way: use proxy_arp to distribute your full ARP table to your clients. Pros: Behaviour is more similar to bridging or LANs. Contra: proxy_arp is considered unsafe, may reveal internal network topology and increases the vulnerability to ARP-based attacks. In addition, it is not IPv6 compatible (there is no neighbour_proxy). - My personal way: Use static ARP and force all traffic (even ''local'' traffic from one domU to another), to be routed via the dom0 as a gateway. Pros: Everything must be routed, and is subject to domain-0''s control. Contra: A little bit harder to setup the domUs if you''ve got local area IP addresses and you''ll need your own vif-scripts. - There are surely other ways, but I''ll not cover that in detail here. Your vif-script should look something like that: # Include common script: . "$(dirname "${0}")/vif-common.sh" # Include vif configuration: . "/etc/xen/network/${vif}.sh" # Switch the command: case "${command}" in "online") # Setup Tun/Tap device: ip link set "${vif}" up echo 1 >"/proc/sys/net/ipv4/conf/${vif}/forwarding" echo 1 >"/proc/sys/net/ipv6/conf/${vif}/forwarding" ip addr add "${gateway_ip}/32" dev "${vif}" # Setup the routes: for addr in ${ip} ; do ip route add "${addr}/32" dev "${vif}" src "${main_ip}" ip neigh replace "${addr}" lladdr "${hw_addr}" nud permanent dev "${vif}" done ;; "offline") # Delete the routes: for addr in ${ip} ; do ip route del "${addr}/32" dev "${vif}" ip neigh del "${addr}" dev "${vif}" done ;; esac # Log success: log debug "Successful vif-myroute ${command} for ${vif}." if [ "${command}" = "online" ] ; then success fi This will require some things (which will make things easier later): - An own configuration per vif in /etc/xen/network, which stores these variables: gateway_ip: The address of the gateway of the ip addresses assigned to the domU. hw_addr: The mac address you assign to the vif (you must set static ones). - To find this file, use unique names for the vifs, e.g. vif-c1 (c for customer) or the names of the domU (can be set by the vifname=... parameter). - You need to specify the ip address of the domU in the vif-configuration or the configuration file above. I hope you are familiar with routing, things will get more complex with policy routing and firewalling. An example setup for the domU: mac address: 00:16:3E:01:02:03 ip address (take a real address here): 123.45.67.89 gateway address (probably specified by the provider, or just take lower address in an as small as possible subnet, that you do *not* really assign to any domUs): 123.45.67.81 (would be gateway of /20 network) There are better way to specify dummy gateway ip addresses, but this makes domU configuration harder (configuration of domU''s OS). Try out this setup and tell me about your problems / successes. The DomU OS would need to configure its ip address statically with the gateway you specified. Regards, Felix Am 13.06.2010 18:20, schrieb Jonathan Tripathy:> Hi Felix, > > Excellent plan! We are getting closer! I should really put all these > wonderful tips from everyone here on a blog or something.. > > Back to the plan.. > > The only difficulty I see here is that the DomUs will be using public > IP address, and the firewall (between the Internet and Dom0) will be a > "filtering bridge" in its own right. However maybe that doesn''t matter. > > Would you maybe be able to give me some example of the actual rules > that I could use? This would be very much appreciated, and if I saw > the rules I could work out if my firewall setup is a problem. > > It would be nice if my ISP just gave my firewall''s WAN interface a > single address, and allowed multiple public subnets to be routed via > my firewall (so my firewall would act like a router, not a bridge), > however I don''t think this is the case. I think all that I will get is > just an ethernet cable connected to a switch.. > > Thanks > > On 13/06/10 17:10, Felix Kuperjans wrote: >> Hi Jonathan, >> >> I read your mail and those you posted in different previous threads and >> I think that you should probably consider *not* using a bridge and using >> pure routing instead: >> - Do you really need bridge-only features (especially broadcasts from >> domU to domU or broadcasts trespassing the dom0)? If I understand your >> plans correctly, you want all your domUs to be isolated with their own >> IP address and only communicating via a dedicated firewall. This way, >> you would not need broadcasts between clients (this is only interesting >> if you want to use LAN services between your domUs, because broadcasts >> are not sent across the internet anyway). >> - AFAIK, routing is more secure and faster than bridging, but somewhat >> harder to setup. >> - You could do what you posted below with routing. It might work with >> bridging, too, but I don''t know a good way to do that with a bridge. >> >> With routing, you would need policy routing because of this elementary >> problem: >> >> You have (to make things easier to explain, in this example only 2) two >> DomUs (let''s say, 1.0.0.1 on vif-1.0 and 1.0.0.2 on vif-2.0), the >> Domain-0 and a dedicated firewall between the Dom0 and the internet. >> If 1.0.0.1 wants to reach any server on the internet (or vice versa), it >> will trespass the firewall by default. >> But if 1.0.0.1 wants to send (e.g. an e-mail) to 1.0.0.2 or (more >> dangerous) wants to attack 1.0.0.2, they would only communicate via the >> Domain-0 (without the firewall). >> >> The problem is: >> >> If you route 1.0.0.2 to vif-2.0 under all circumstances, it will bypass >> the firewall if 1.0.0.1 sent the package. >> If you route everything except 1.0.0.1 via eth0, you wont be able to >> reach 1.0.0.2 any way. >> >> The solution is: >> >> You need to do policy routing. >> If a package originates from the internet an should be sent to 1.0.0.2, >> it must be routed to vif-2.0. But if it originates from 1.0.0.1, it must >> be routed to eth0, so that it is sent to the firewall. >> The firewall will then process the package and return it to the server, >> which now must route the package to vif-2.0. >> >> So it will take two policy routes: >> route 1.0.0.2 via vif-2.0, if it is from eth0 >> route 1.0.0.2 via eth0, if it is not from eth0 >> >> I don''t think that those routes would work with a bridge, so consider >> using routing. >> >> Felix >> >> Am 13.06.2010 17:45, schrieb Jonathan Tripathy: >> >>> Hi Everyone, >>> >>> Does anyone know any rules that I could use (using iptable, ebtables, >>> or otherwise) that could force all traffic coming from a guest to go >>> out via a particular interface? I wish to stop "inter-guest" >>> communication, without going via my firewall first. >>> >>> Thanks >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >>> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users