Xen users - Feb 2013 - Question regarding xen networking (bonding, xen-bridging)

Hello,

I fail to fully understand the xen bridge networking concept. In 
particular, the relation between bonding, bridging and Vlans is not 
clear to me. I have to admit that I''m not a network/bridging/routing 
expert at all. I''ll try to explain setup and problem in detail. Please 
apologize if anything sounds illegitimate or absurd to you, or if I 
missed important information. My system is a Debian/Wheezy

Usually, we use a setup with different Vlan tags for each domU. 
Eth0/bond0 on the dom0 are untagged. For every domU, we create a 
interface ''vlanXXX'' with eth0/bond0 as raw device, and a
bridge ''brXXX''
with the vlan interface as bridge port. brXXX is the bridge used for the 
domU:

auto vlanXXX
iface vlanXXX inet manual
     vlan_raw_device bond0

auto brXXX
iface brXXX inet manual
     bridge_ports vlanXXX
     bridge_stp off
     bridge_fd 0

This works quite well. But now I tried (and until now failed) to setup 
a Xen host with bridging, and dom0 + domUs in the same network (no Vlan 
management on dom0). After quite some time I got it working half the 
way. The bond0 device is set up as manual, and a bridge
''xenbr0'' is
created with interface ''bond0'' as bridge port. The domU uses
''xenbr0'' as
bridge:

  On the dom0 LAN connection works as expected. Gateway responds, 
connection to the internet works. Dom0 and domU see each other.

But from domU, everything beyond the dom0 is unreachable, e.g. the 
gateway doesn''t respond. MAC address from the domU is propagated to 
switches and gateway, I can see it in the arp table. In other words, the 
packets from domU find their way out, but the responding packets don''t 
find their way back. A quick look at the iptables rules on dom0 give me 
the impression, that dom0 doesn''t know how to handle packets for domU:

# iptables -L FORWARD
Chain FORWARD (policy ACCEPT)
target   prot opt source     destination
ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out 
vif1.0 --physdev-is-bridged
ACCEPT   udp  --  anywhere   anywhere   PHYSDEV match --physdev-in 
vif1.0 --physdev-is-bridged udp spt:bootpc dpt:bootps
ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out 
vif1.0 --physdev-is-bridged
ACCEPT   all  --  <DOMU-IP>  anywhere   PHYSDEV match --physdev-in 
vif1.0 --physdev-is-bridged


The last rule seems to accept packets with domU-IP as source, but I 
cannot find a rule which handles incoming packets for domU. In other 
words, the dom0 doesn''t know what to do with packets for the domU. Is
my
assumption correct? If yes, why is this the case? Is it related to 
interface bonding?

I don''t get the picture, maybe you can help me :)

Kind regards,
  jonas

Jonas Meurer wrote:
>  On the dom0 LAN connection works as expected. Gateway responds, 
>connection to the internet works. Dom0 and domU see each other.
>
>But from domU, everything beyond the dom0 is unreachable, e.g. the 
>gateway doesn''t respond. MAC address from the domU is propagated to
>switches and gateway, I can see it in the arp table. In other words, the 
>packets from domU find their way out, but the responding packets
don''t
>find their way back. A quick look at the iptables rules on dom0 give me 
>the impression, that dom0 doesn''t know how to handle packets for
domU:
>
># iptables -L FORWARD
>Chain FORWARD (policy ACCEPT)
>target   prot opt source     destination
>ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out 
>vif1.0 --physdev-is-bridged
>ACCEPT   udp  --  anywhere   anywhere   PHYSDEV match --physdev-in 
>vif1.0 --physdev-is-bridged udp spt:bootpc dpt:bootps
>ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out 
>vif1.0 --physdev-is-bridged
>ACCEPT   all  --  <DOMU-IP>  anywhere   PHYSDEV match --physdev-in 
>vif1.0 --physdev-is-bridged
>
>
>The last rule seems to accept packets with domU-IP as source, but I 
>cannot find a rule which handles incoming packets for domU. In other 
>words, the dom0 doesn''t know what to do with packets for the domU.
Is my
>assumption correct? If yes, why is this the case? Is it related to 
>interface bonding?
I think you''ve answered your own question there. For testing purposes,
clear iptables and see if everything starts working.

There is something in the standard scripts that is supposed to manipulate
iptables rules when guest domains are started/stopped (there is the option of
specifying "ip-a.b.c.d" in the VIF declaration), but I''ve
never looked into it.

Am 18.02.2013 13:33, schrieb Simon Hobson:
> Jonas Meurer wrote:
>
>>  On the dom0 LAN connection works as expected. Gateway responds,
>>connection to the internet works. Dom0 and domU see each other.
>>
>>But from domU, everything beyond the dom0 is unreachable, e.g. the
>>gateway doesn''t respond. MAC address from the domU is
propagated to
>>switches and gateway, I can see it in the arp table. In other words, 
>> the
>>packets from domU find their way out, but the responding packets 
>> don''t
>>find their way back. A quick look at the iptables rules on dom0 give 
>> me
>>the impression, that dom0 doesn''t know how to handle packets
for
>> domU:
>>
>># iptables -L FORWARD
>>Chain FORWARD (policy ACCEPT)
>>target   prot opt source     destination
>>ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out
>>vif1.0 --physdev-is-bridged
>>ACCEPT   udp  --  anywhere   anywhere   PHYSDEV match --physdev-in
>>vif1.0 --physdev-is-bridged udp spt:bootpc dpt:bootps
>>ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out
>>vif1.0 --physdev-is-bridged
>>ACCEPT   all  --  <DOMU-IP>  anywhere   PHYSDEV match --physdev-in
>>vif1.0 --physdev-is-bridged
>>
>>
>>The last rule seems to accept packets with domU-IP as source, but I
>>cannot find a rule which handles incoming packets for domU. In other
>>words, the dom0 doesn''t know what to do with packets for the
domU. Is
>> my
>>assumption correct? If yes, why is this the case? Is it related to
>>interface bonding?
>
> I think you''ve answered your own question there. For testing
> purposes, clear iptables and see if everything starts working.
>
> There is something in the standard scripts that is supposed to
> manipulate iptables rules when guest domains are started/stopped
> (there is the option of specifying "ip-a.b.c.d" in the VIF
> declaration), but I''ve never looked into it.
Hello Simon,

Thanks for the quick response. For some unknown reason, suddenly the 
setup started to work. Maybe I didn''t wait long enough for the switches
to accept the new domU as devices. Whatever, below is the dom0 interface 
setup that works for me (as reference for the future):

/etc/network/interfaces:

--8-<----8-<----8-<----8-<----8-<----8-<----8-<--
auto bond0
iface bond0 inet manual
         slaves eth0 eth1
         bond_mode balance-xor
         bond_miimon 100
         bond_downdelay 200
         bond_updelay 200

auto xenbr0
iface xenbr0 inet static
         bridge_ports bond0
         bridge_maxwait 0
         bridge_stp off
         bridge_fd 0
         address <IP-ADDRESS>
         netmask <IP-NETMASK>
         network <IP-NETWORK>
         broadcast <IP-BROADCAST>
         gateway <IP-GATEWAY>
-->-8---->-8---->-8---->-8---->-8---->-8---->-8--

I didn''t modify the iptables rules created by xen helper scripts:

# iptables-save

--8-<----8-<----8-<----8-<----8-<----8-<----8-<--
-A FORWARD -m physdev --physdev-out vif2.0 --physdev-is-bridged -j 
ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif2.0 --physdev-is-bridged 
-m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -m physdev --physdev-out vif2.0 --physdev-is-bridged -j 
ACCEPT
-A FORWARD -s <DOMU-IP>/32 -m physdev --physdev-in vif2.0 
--physdev-is-bridged -j ACCEPT
-->-8---->-8---->-8---->-8---->-8---->-8---->-8--

Kind regards,
  jonas

Hello.

El 18/02/13 05:47, Jonas Meurer escribió:
>   On the dom0 LAN connection works as expected. Gateway responds,
> connection to the internet works. Dom0 and domU see each other.
Make sure, the DomU''s interface is really attached to the intended
bridge.
xm list
brctl show
> But from domU, everything beyond the dom0 is unreachable, e.g. the
> gateway doesn''t respond. MAC address from the domU is propagated
to
> switches and gateway, I can see it in the arp table. In other words, the
> packets from domU find their way out, but the responding packets
don''t
> find their way back.
Double check your routing and ipfilter (or whatever) configuration on 
your DomU and the gateway. This is the most common cause of issues with 
this description.

> A quick look at the iptables rules on dom0 give me
> the impression, that dom0 doesn''t know how to handle packets for
domU:
>
> # iptables -L FORWARD
> Chain FORWARD (policy ACCEPT)
> target   prot opt source     destination
> ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out
> vif1.0 --physdev-is-bridged
> ACCEPT   udp  --  anywhere   anywhere   PHYSDEV match --physdev-in
> vif1.0 --physdev-is-bridged udp spt:bootpc dpt:bootps
> ACCEPT   all  --  anywhere   anywhere   PHYSDEV match --physdev-out
> vif1.0 --physdev-is-bridged
> ACCEPT   all  --  <DOMU-IP>  anywhere   PHYSDEV match --physdev-in
> vif1.0 --physdev-is-bridged
>
> The last rule seems to accept packets [...]
Your policy is ACCEPT. So, unless there is a specific DROP or REJECT 
rule, you may count on netfilter does not prevent the packets from arriving.
> with domU-IP as source, but I
> cannot find a rule which handles incoming packets for domU.
That is a, mostly, antyspoofing rule. Most of Xen''t iptabels setup is 
intended to protect the network from the DomU''s, so it does not filter 
incoming traffic by default.
> In other
> words, the dom0 doesn''t know what to do with packets for the domU.
Is my
> assumption correct?
It does not need to know a too much. It a packet made it to the bridge, 
it should be picked up by the interface (member of the bridge) with the 
corresponding destination MAC. That is, unless there is some filtering 
preventing it, which seems not to be the case.

Check with tcpdump on Dom0 (bridge) and DomU (interface). You are 
supposed to see the same traffic involving your DomU''s MAC address, 
including the incoming responses to outgoing connections. Any difference 
should bring some light to the matter.
> Is it related to interface bonding?
Should not. A incorrect bonding setup could prevent packets leaving 
physical interface, or arriving on it. If a packet is visible on the 
bridge, than it already has made it through the bond, so we may assume 
it works fine, unless it looses some packets.


-- 
Alexandre Kouznetsov