Hi, I''m currently switching from toolstacks from xm to xl. Because I have a rather complex network setup with my domUs, I use xen''s nat capabilities. But what works fine with xm behaves slightly strange in xl: I use something like the following to establish a domU interface within my vif part: ''type=vif, mac=00:16:3E:06:DA:B2, ip=192.168.2.1, vifname=fw11, model=e1000, script=vif-nat'' Now xm would create an interface called ''fw11'' with ip 192.168.2.1 and netmask 255.255.255.0. I don''t really care about the rest (iptables, routes) because I overwrite this from other places anyway.. But xl creates an interface ''fw11'' with ip 192.168.2.128 and netmask 255.255.255.255 which is defenetly not what I was expecting. Can somebody explain why xl behaves the way it does and how can I change this? I could only find vif-nat setting routes but nothing about netmasks and IPs so I guess i''m missing something here.. I''m using the latest xen-unstable and a vanilla 3.7.4 kernel.. Thanks!
On Mon, 2013-01-28 at 01:28 +0000, Matthias wrote:> Hi, > > I''m currently switching from toolstacks from xm to xl. Because I have > a rather complex network setup with my domUs, I use xen''s nat > capabilities. But what works fine with xm behaves slightly strange in > xl: > > I use something like the following to establish a domU interface > within my vif part: > > ''type=vif, mac=00:16:3E:06:DA:B2, ip=192.168.2.1, vifname=fw11, > model=e1000, script=vif-nat'' > > Now xm would create an interface called ''fw11'' with ip 192.168.2.1 and > netmask 255.255.255.0. I don''t really care about the rest (iptables, > routes) because I overwrite this from other places anyway.. > > But xl creates an interface ''fw11'' with ip 192.168.2.128 and netmask > 255.255.255.255 which is defenetly not what I was expecting.Absolutely! At one point xl had a bug with handling spaces in the vif specification, which might cause ip= to not take affect. I thought it was fixed, but perhaps not? Do you see an "ip" node under the vif backend device in xenstore? Can you try dropping the spaces in your configuration file?> Can somebody explain why xl behaves the way it does and how can I > change this? I could only find vif-nat setting routes but nothing > about netmasks and IPs so I guess i''m missing something here..I expect that the .2.128 comes from routing_ip() in the vif-nat script. However this hasn''t changed recently and should behave the same under xl as it did under xm so obviously something is up. If it''s not the problem above then you might need to add some additional logging to the vif-nat script to figure out why it isn''t accepting your ip setting. Ian.
Hi,
I finally got the chance to do the testing..
You were right, the issue was due to routing_ip or better why the
calculated ($ip + 127) is used later here:
do_or_die ip addr add "$router_ip" dev "${dev}"
do_or_die ip route add "$vif_ip" dev "${dev}" src
"$router_ip"
I don''t really see any point in creating a dedicated router-IP and
using it this way and i dont'' see why this worked for me when using xm
earlier, but I might did some hacking before to fix this in my xm
installation, i can''t really remember..
At the end of the day, I did the following to fix the behavior for my
installation: (it''s only exchanging router_ip to vif_ip)
diff vif-nat vif-nat_backup
174,177c174,175
< #do_or_die ip addr add "$router_ip" dev
"${dev}"
< do_or_die ip addr add "$vif_ip" dev "${dev}"
< #do_or_die ip route add "$vif_ip" dev "${dev}"
src "$router_ip"
< do_or_die ip route add "$vif_ip" dev "${dev}" src
"$vif_ip"
---> do_or_die ip addr add "$router_ip" dev "${dev}"
> do_or_die ip route add "$vif_ip" dev "${dev}"
src "$router_ip"
This way i get correctly named interfaces with the right IPs to use in
iptables and policy based routing tables..
2013/1/28 Ian Campbell <Ian.Campbell@citrix.com>:> On Mon, 2013-01-28 at 01:28 +0000, Matthias wrote:
>> Hi,
>>
>> I''m currently switching from toolstacks from xm to xl. Because
I have
>> a rather complex network setup with my domUs, I use xen''s nat
>> capabilities. But what works fine with xm behaves slightly strange in
>> xl:
>>
>> I use something like the following to establish a domU interface
>> within my vif part:
>>
>> ''type=vif, mac=00:16:3E:06:DA:B2, ip=192.168.2.1,
vifname=fw11,
>> model=e1000, script=vif-nat''
>>
>> Now xm would create an interface called ''fw11'' with
ip 192.168.2.1 and
>> netmask 255.255.255.0. I don''t really care about the rest
(iptables,
>> routes) because I overwrite this from other places anyway..
>>
>> But xl creates an interface ''fw11'' with ip
192.168.2.128 and netmask
>> 255.255.255.255 which is defenetly not what I was expecting.
>
> Absolutely!
>
> At one point xl had a bug with handling spaces in the vif specification,
> which might cause ip= to not take affect. I thought it was fixed, but
> perhaps not?
>
> Do you see an "ip" node under the vif backend device in xenstore?
>
> Can you try dropping the spaces in your configuration file?
>
>> Can somebody explain why xl behaves the way it does and how can I
>> change this? I could only find vif-nat setting routes but nothing
>> about netmasks and IPs so I guess i''m missing something here..
>
> I expect that the .2.128 comes from routing_ip() in the vif-nat script.
> However this hasn''t changed recently and should behave the same
under xl
> as it did under xm so obviously something is up.
>
> If it''s not the problem above then you might need to add some
additional
> logging to the vif-nat script to figure out why it isn''t accepting
your
> ip setting.
>
> Ian.
>
On Sun, 2013-02-03 at 21:29 +0000, Matthias wrote:> Hi, > > I finally got the chance to do the testing.. > > You were right, the issue was due to routing_ip or better why the > calculated ($ip + 127) is used later here: > > do_or_die ip addr add "$router_ip" dev "${dev}" > do_or_die ip route add "$vif_ip" dev "${dev}" src "$router_ip" > > I don''t really see any point in creating a dedicated router-IP andGlad you''ve got it sorted, I don''t understand this stuff either though I''m afraid. (Perhaps they form two ends of a point-to-point /31 network? Total guess)> using it this way and i dont'' see why this worked for me when using xm > earlier, but I might did some hacking before to fix this in my xm > installation, i can''t really remember..FWIW this script doesn''t appear to have changed in hg for many years. Ian.
I guess i finally understood the basic idea of the nat script:
You set an IP for the vif interface. This will not be the IP of the
interface, but the IP the domU will be using internally. This way if
the dom0 receives a package for the configured IP, it will be routed
through the dom0-interface to the domU. (In detail: This way the
package goes into the iptables forward chain and not the input chain,
changing the hosts general treatment of the package).
So not setting the interface to the configured IP but only creating an
ip route makes perfect sense.
But then the vif-nat script assignes an IP+127 to the interface. This
creates 2 problems in my opinion:
1) When using the dom0 for routing network traffic from one domU to an
other, so basically using the dom0 as a network cable, the src
attribute used in the routing creation will alter the source ip
address from one domU to the ip+127 address which is, at least in my
network setup, pretty annoying.
So if you got domUs A (IP 10.0.0.1) and B (IP 10.0.0.2) and send
traffic from A to B, B will see the traffic coming from 10.0.0.128
instead of .1 .
2) I haven''t tested it, but I don''t really know what would
happen if
you set IPs >128 in the domU configuration and this way create
out-of-bounce routing IPs.
Well, I think that most people don''t use the nat feature of xen and
this isn''t tested much.. But somebody might want to look at it at a
given point of time...
For me, changing the IP setting lines to
do_or_die ip addr add "0.0.0.0" dev "${dev}"
do_or_die ip route add "$vif_ip" dev "${dev}"
This doesn''t set an IP to the interface in the domU and no src.
attribute to the route, fixing the above issues.. Might be worth
thinkin about including in xen main line, i don''t really know..
Posted this only if somebody might need this later..
2013/2/4 Ian Campbell <Ian.Campbell@citrix.com>:> On Sun, 2013-02-03 at 21:29 +0000, Matthias wrote:
>> Hi,
>>
>> I finally got the chance to do the testing..
>>
>> You were right, the issue was due to routing_ip or better why the
>> calculated ($ip + 127) is used later here:
>>
>> do_or_die ip addr add "$router_ip" dev "${dev}"
>> do_or_die ip route add "$vif_ip" dev "${dev}" src
"$router_ip"
>>
>> I don''t really see any point in creating a dedicated router-IP
and
>
> Glad you''ve got it sorted, I don''t understand this stuff
either though
> I''m afraid. (Perhaps they form two ends of a point-to-point /31
network?
> Total guess)
>
>> using it this way and i dont'' see why this worked for me when
using xm
>> earlier, but I might did some hacking before to fix this in my xm
>> installation, i can''t really remember..
>
> FWIW this script doesn''t appear to have changed in hg for many
years.
>
> Ian.
>
On Mon, 2013-02-04 at 21:28 +0000, Matthias wrote:> I guess i finally understood the basic idea of the nat script: > > You set an IP for the vif interface. This will not be the IP of the > interface, but the IP the domU will be using internally. This way if > the dom0 receives a package for the configured IP, it will be routed > through the dom0-interface to the domU. (In detail: This way the > package goes into the iptables forward chain and not the input chain, > changing the hosts general treatment of the package). > > So not setting the interface to the configured IP but only creating an > ip route makes perfect sense.Is an IP on the vifX.Y device not required in order to route to it? And does that IP not need to differ from the one on the guests eth0 device? (otherwise the packet will get delivered to dom0) In the routing variant we use the IP address of dom0''s own interface on all of the vifX.Y devices, I wonder why this doesn''t suffice for NAT.> But then the vif-nat script assignes an IP+127 to the interface. This > creates 2 problems in my opinion: > > 1) When using the dom0 for routing network traffic from one domU to an > other, so basically using the dom0 as a network cable, the src > attribute used in the routing creation will alter the source ip > address from one domU to the ip+127 address which is, at least in my > network setup, pretty annoying. > > So if you got domUs A (IP 10.0.0.1) and B (IP 10.0.0.2) and send > traffic from A to B, B will see the traffic coming from 10.0.0.128 > instead of .1 .Isn''t this an expected property of NAT? When communicating with off box hosts/VMs the IP address would be dom0''s IP address. Sounds like you might actually want a nat''ed brouter type configuration? (i.e. the guests are bridged but offbox traffic is nat''ed)> 2) I haven''t tested it, but I don''t really know what would happen if > you set IPs >128 in the domU configuration and this way create > out-of-bounce routing IPs.I suspect this is probably a case of "don''t do that".> Well, I think that most people don''t use the nat feature of xen and > this isn''t tested much.. But somebody might want to look at it at a > given point of time...It''s certainly not the most widely used configuration.> For me, changing the IP setting lines to > > do_or_die ip addr add "0.0.0.0" dev "${dev}" > do_or_die ip route add "$vif_ip" dev "${dev}" > > This doesn''t set an IP to the interface in the domU and no src. > attribute to the route, fixing the above issues.. Might be worth > thinkin about including in xen main line, i don''t really know..Me neither.> Posted this only if somebody might need this later..Thanks. Ian.