Xen users - Jul 2009 - Fully virtualized domU Linux network (VLAN) trouble

Hi,

I have a 64bit Debian Lenny server with two physical network interfaces,
one is connected to a switch without VLANs (eth0), the other (eth1) is
connected to a switch with tagged VLANs (eth1). I''m running a couple of
paravirtualized VMs that have no problem using the eth0 interface
(bridged). I''m also running a couple of fully virtualized VMs, one
being
a Linux box (with a specific kernel); others being Windows VMs with a
single network interface bridged to eth0. The Windows VMs are working
fine. The (fully virtualized) Linux VM has two network interfaces
bridged to the physical network interfaces in dom0 (ie. to both eth0 and
eth1). This VM comes up fine, has two network interfaces as specified in
its Xen config but is not able to use any of the networks on eth0 and
the VLANs on eth1. The VM obviously has the vlan package installed and
the VLAN interfaces are set up.

So: all VMs (para Linux and full Windows) except the fully virtualized
Linux VM are able to use the eth0 interface (no VLANs). The fully
virtualized Linux VM is not able to use eth0 (no VLANs), nor eth1 (with
tagged VLANs).

I''ve managed to get it to work with all interfaces once (ie. eth0 and a
few VLAN interfaces on eth1), then had to change the network config and
lost all network connectivity after the reboot. I''ve also observed that
sometimes one virtual nic can be used, sometimes the other. The only
variable here being making changes to the vif entry in the Xen config
file for this VM. It currently looks like:

vif        = [''mac=03:16:3e:00:02:39, type=ioemu,
bridge=eth0'',
           ''mac=03:16:3E:01:00:00, type=ioemu, bridge=eth1'']

The vif interfaces are created and bridged correctly in the dom0 once
the fully virtualized Linux domU starts.

dom0 has no business on the VLANs so it seems preferable to setup VLANs
in domU, not dom0. I''ve found a few documents that suggest to use VLANs
in dom0, and bridge the virtual interfaces to the VLAN interfaces in
dom0. I''ve spend a few hours trying to get that to work but
unfortunately that didn''t work either.

The physical NICs are:
00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)

How should I further debug/analyse this problem?

kind regards,
Jan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

After some more experiments I have now concluded that using VLANs in a
fully virtualized domU doesn''t seem to work properly. It works fine if
domU is para virtualized. Even though I managed to get it to work once,
a change in the config file resulted in Xen generating new MAC addresses
and the subsequent updates to the xenstore db rendered the VM inoperable
(with its network) again. At least that''s my conclusion so far.

Does anyone have any success with VLANs in fully virtualized domU''s?

Debian Lenny 64 bit, Xen 3.2.1, kernel 2.6.26-2-xen-amd64, Super micro
hardware, Quad core AMD, NVIDIA MCP55 NICs.

best,
Jan


Jan Bakuwel wrote:
> Hi,
>
> I have a 64bit Debian Lenny server with two physical network interfaces,
> one is connected to a switch without VLANs (eth0), the other (eth1) is
> connected to a switch with tagged VLANs (eth1). I''m running a
couple of
> paravirtualized VMs that have no problem using the eth0 interface
> (bridged). I''m also running a couple of fully virtualized VMs, one
being
> a Linux box (with a specific kernel); others being Windows VMs with a
> single network interface bridged to eth0. The Windows VMs are working
> fine. The (fully virtualized) Linux VM has two network interfaces
> bridged to the physical network interfaces in dom0 (ie. to both eth0 and
> eth1). This VM comes up fine, has two network interfaces as specified in
> its Xen config but is not able to use any of the networks on eth0 and
> the VLANs on eth1. The VM obviously has the vlan package installed and
> the VLAN interfaces are set up.
>
> So: all VMs (para Linux and full Windows) except the fully virtualized
> Linux VM are able to use the eth0 interface (no VLANs). The fully
> virtualized Linux VM is not able to use eth0 (no VLANs), nor eth1 (with
> tagged VLANs).
>
> I''ve managed to get it to work with all interfaces once (ie. eth0
and a
> few VLAN interfaces on eth1), then had to change the network config and
> lost all network connectivity after the reboot. I''ve also observed
that
> sometimes one virtual nic can be used, sometimes the other. The only
> variable here being making changes to the vif entry in the Xen config
> file for this VM. It currently looks like:
>
> vif        = [''mac=03:16:3e:00:02:39, type=ioemu,
bridge=eth0'',
>            ''mac=03:16:3E:01:00:00, type=ioemu,
bridge=eth1'']
>
> The vif interfaces are created and bridged correctly in the dom0 once
> the fully virtualized Linux domU starts.
>
> dom0 has no business on the VLANs so it seems preferable to setup VLANs
> in domU, not dom0. I''ve found a few documents that suggest to use
VLANs
> in dom0, and bridge the virtual interfaces to the VLAN interfaces in
> dom0. I''ve spend a few hours trying to get that to work but
> unfortunately that didn''t work either.
>
> The physical NICs are:
> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>
> How should I further debug/analyse this problem?
>
> kind regards,
> Jan
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>   

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Jul 28, 2009 at 8:56 PM, Jan Bakuwel<jan.bakuwel@gmail.com>
wrote:
> I''m also running a couple of fully virtualized VMs, one being
> a Linux box (with a specific kernel);
Which kernel version?
Why do you use HVM guest? You''d generally get much better performance
with PV guests.
Also, if you absolutely have to use HVM guest, you''d get better
performance if you use PV drivers (which AFAIK, only works on 2.6.18
kernel).
> vif        = [''mac=03:16:3e:00:02:39, type=ioemu,
bridge=eth0'',
>           ''mac=03:16:3E:01:00:00, type=ioemu,
bridge=eth1'']
If you use PV drivers, you should remove "type=ioemu".
> How should I further debug/analyse this problem?
I''d start with trying to get HVM domU''s traffic work
correctly,
preferably with PV drivers. Depending on your kernel version, it might
involve changing domU kernel. Once that work, start with the vlans.
One of the things to check is
/proc/sys/net/bridge/bridge-nf-filter-vlan-tagged

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Jul 31, 2009 at 7:09 AM, Jan Bakuwel<jan.bakuwel@gmail.com>
wrote:
>> Also, if you absolutely have to use HVM guest, you''d get
better
>> performance if you use PV drivers (which AFAIK, only works on 2.6.18
>> kernel).
>>
>
> There are PV drivers for Linux as well? I thought only for Windows?
Yes, there''s also PV drivers for Linux.It''s (ironically)
HARDER to use
compared to James Harper''s GPLPV for Windows, as the Linux version:
- only works for 2.6.18
- adds new interface and (possibly) block device
- requires certain kernel boot parameter and rebuilding initrd to have
block device works correctly
>
>> I''d start with trying to get HVM domU''s traffic work
correctly,
>> preferably with PV drivers. Depending on your kernel version, it might
>> involve changing domU kernel. Once that work, start with the vlans.
>> One of the things to check is
>> /proc/sys/net/bridge/bridge-nf-filter-vlan-tagged
>>
>
> I''ve meanwhile discovered it must be something to do with the
Linux HVM
> drivers; when running the same userland with the Xen kernel (ie.
> paravirtualised), all works as expected.
Well, in that case you might find some luck using another kernel,
possibly compiling your own :)

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users