> I am setting up a box with FC6 + xen with FC6 for the DomU guests. > I have a server with 4 gigabit ports and I want to use 802.1Q vlans on > one of the gig ports. > I have the VLANs configured in Dom0 without any addresses attached to > them. > How do I make the VLANs available to the guests? I want to make 4 > VLANs available to each guest. I have thought of a few alternatives > but I cannot get any to work.The easy way; I have installed FC6 with 802.1Q and 4 guests. It works very nicely. For each guest do: 1 - You need to enable vlans /etc/sysconfig/network. NETWORKING=yes VLAN=yes NETWORKING_IPV6=no HOSTNAME=xenbox2.example.com GATEWAY=192.168.42.1 2 - create a subinterface by file editting /etc/sysconfig/network-scripts/ifcfg-eth0.42: DEVICE=eth0.42 BOOTPROTO=static ONBOOT=yes IPADDR=192.168.42.102 NETMASK=255.255.255.0 TYPE=Ethernet 3 - depending on whether you want untagged packets on the same interface, you need to edit the /etc/.../ifcfg-eth0 file too. (note: the parent interface must be up for the tagged interface to work.) The lines commented out will assign an IP address to the untagged interface: DEVICE=eth0 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet # BOOTPROTO=static # IPADDR=172.16.42.102 # NETMASK=255.255.255.0 4 - make sure that your /etc/hosts setup is correct. If you have DNS available 24/7, then only define localhost[.localdomain} as 127.0.0.1 and not the hostname itself. When you restart the network in the guest, then your guest will default to use the tagged interface vlan 42 (above example). If you want the untagged interface as default, then you can change the gateway and assign an IP address to it. On the host, you need to do a similar thing. Create a subinterface eth0.42 and assign an IP address to it. Once that is done, you can talk to your guests via dot1Q from your host. Beware of iptables! If you have a firewall setup, then you need to make sure that it is done right. Doing it right, though, is a though job with dot1Q in place. I have the firewall completely disabled on the host and guests. My firewall is placed physically separated on another box.> 1. Bridge the tagged 802.1Q stream from the Gig port through to the > guests and configure eth0.x. on the guestsThis would be above scenario.> 2. Create a bridge on Dom0 for each VLAN so that they appear as > eth0...eth3 on the guests.The xenbrX interfaces on the host already transport all packets because they are L2 bridges (see brctl). The bridge does not care whether the packets are .1Q tagged or not. The only thing you need is to instruct the kernel to look for/act on specifics by creating (sub-)interfaces. Now, if you want the guests not to know that they are connected to a VLAN, then you need a brigde-interface to do (un-)tagging for you. That means that you need to create a dot1Q subinterface on the host''s physical peth[0-3] and add the subinterface to a new bridge. Normal scenario: peth0 -> xenbr0 -> vif0.0(host eth0) vif1.0(guest1 eth0) Dot1Q scenario: peth0.42 -> xenbr42 -> vif1.0(guest1 eth0) i.e.: # brctl addbr xenbr42 # ip link set xenbr42 up # vconfig add peth0 42 # ip link set peth0.42 up # brctl addif xenbr42 peth0.42 In the VM config you have now something like: vif = [ ''mac=00:16:3e:11:22:33, bridge=xenbr42'', ] The VM''s eth0 interface is created as part of bridge that will fowrard to a subinterface. The kernel will now do the tagging/untagging (note: slow) and the VM cannot see that it is attached to a VLAN. Theoretically it should work, but I was not able to make this scenario work as expected though. I suspect that the hosts peth0 -> eth0 bridge is interfering (which has a tagged interface too in my setup). Or maybe my fingers were too fast at typing and I forgot a simple thing...> The xen documentation is minimal for configuring VLANs. Is there a > (FC6) supported way of configuring them?Vlans is a no-issue for xen. It is layered in the network and xen only passes the packets. -- Greetings Bertho Bertho Stultiens Senior Systems Manager Mobilethink A/S
Thanks Bertho,
I managed to get it to work something along the lines of your first
method before I read your email. I was also unsuccessful using your
second method.
I wanted a Dom0 host to be accessable only on an administration network
with the guests accessing their relevant VLANs
I used one Gig (eth0) real interface as an 802.1Q trunk with the Dom0
having no interfaces mapped to it.
The Dom0 was connected (without VLANS) to the administration network via
eth1. I had to modify /etc/xen/xend-config.sxp to bridge eth0 otherwise
it would bridge the active device (eth1)
eth0 --> xenbr0 -> Dom1 --> VLAN1
|-> VLAN2
|->...
eth1-->Dom0
(network-script ''network-bridge netdev=eth0'')
The guests were then set up as normal hosts connected to an 802.1Q trunk
network as you detailed.
I found that if I used VLANs on the Dom0 on eth0 then the trunk would
not pass through to the guests, Hence why Dom0 is not using VLANs on eth0.
I did not do much research to discover why except that the instant I
removed the VLANs from Dom0 eth0 the VLAN networks started working on
the guest.
My initial email came about because I was expecting to have to plumb the
individual VLANs through to the guests as you also failed to do. This is
much like VMWare ESX works.
Regards
Darryl
Bertho Stultiens wrote:>> I am setting up a box with FC6 + xen with FC6 for the DomU guests.
>> I have a server with 4 gigabit ports and I want to use 802.1Q vlans on
>> one of the gig ports.
>> I have the VLANs configured in Dom0 without any addresses attached to
>> them.
>> How do I make the VLANs available to the guests? I want to make 4
>> VLANs available to each guest. I have thought of a few alternatives
>> but I cannot get any to work.
>>
>
> The easy way; I have installed FC6 with 802.1Q and 4 guests. It works
> very nicely. For each guest do:
> 1 - You need to enable vlans /etc/sysconfig/network.
> NETWORKING=yes
> VLAN=yes
> NETWORKING_IPV6=no
> HOSTNAME=xenbox2.example.com
> GATEWAY=192.168.42.1
>
> 2 - create a subinterface by file editting
> /etc/sysconfig/network-scripts/ifcfg-eth0.42:
> DEVICE=eth0.42
> BOOTPROTO=static
> ONBOOT=yes
> IPADDR=192.168.42.102
> NETMASK=255.255.255.0
> TYPE=Ethernet
>
> 3 - depending on whether you want untagged packets on the same
> interface, you need to edit the /etc/.../ifcfg-eth0 file too. (note: the
> parent interface must be up for the tagged interface to work.) The lines
> commented out will assign an IP address to the untagged interface:
> DEVICE=eth0
> BOOTPROTO=none
> ONBOOT=yes
> TYPE=Ethernet
> # BOOTPROTO=static
> # IPADDR=172.16.42.102
> # NETMASK=255.255.255.0
>
> 4 - make sure that your /etc/hosts setup is correct. If you have DNS
> available 24/7, then only define localhost[.localdomain} as 127.0.0.1
> and not the hostname itself.
>
> When you restart the network in the guest, then your guest will default
> to use the tagged interface vlan 42 (above example). If you want the
> untagged interface as default, then you can change the gateway and
> assign an IP address to it.
>
> On the host, you need to do a similar thing. Create a subinterface
> eth0.42 and assign an IP address to it. Once that is done, you can talk
> to your guests via dot1Q from your host.
>
> Beware of iptables! If you have a firewall setup, then you need to make
> sure that it is done right. Doing it right, though, is a though job with
> dot1Q in place. I have the firewall completely disabled on the host and
> guests. My firewall is placed physically separated on another box.
>
>
>
>> 1. Bridge the tagged 802.1Q stream from the Gig port through to the
>> guests and configure eth0.x. on the guests
>>
>
> This would be above scenario.
>
>
>
>> 2. Create a bridge on Dom0 for each VLAN so that they appear as
>> eth0...eth3 on the guests.
>>
>
> The xenbrX interfaces on the host already transport all packets because
> they are L2 bridges (see brctl). The bridge does not care whether the
> packets are .1Q tagged or not. The only thing you need is to instruct
> the kernel to look for/act on specifics by creating (sub-)interfaces.
>
> Now, if you want the guests not to know that they are connected to a
> VLAN, then you need a brigde-interface to do (un-)tagging for you. That
> means that you need to create a dot1Q subinterface on the host''s
> physical peth[0-3] and add the subinterface to a new bridge.
>
> Normal scenario:
> peth0 -> xenbr0 -> vif0.0(host eth0)
> vif1.0(guest1 eth0)
>
> Dot1Q scenario:
> peth0.42 -> xenbr42 -> vif1.0(guest1 eth0)
>
> i.e.:
> # brctl addbr xenbr42
> # ip link set xenbr42 up
> # vconfig add peth0 42
> # ip link set peth0.42 up
> # brctl addif xenbr42 peth0.42
>
> In the VM config you have now something like:
> vif = [ ''mac=00:16:3e:11:22:33, bridge=xenbr42'', ]
>
> The VM''s eth0 interface is created as part of bridge that will
fowrard
> to a subinterface. The kernel will now do the tagging/untagging (note:
> slow) and the VM cannot see that it is attached to a VLAN.
>
> Theoretically it should work, but I was not able to make this scenario
> work as expected though. I suspect that the hosts peth0 -> eth0 bridge
> is interfering (which has a tagged interface too in my setup). Or maybe
> my fingers were too fast at typing and I forgot a simple thing...
>
>
>
>
>> The xen documentation is minimal for configuring VLANs. Is there a
>> (FC6) supported way of configuring them?
>>
>
> Vlans is a no-issue for xen. It is layered in the network and xen only
> passes the packets.
>
>
>
__________________________________________________________________________________
DISCLAIMER
The contents of this electronic message and any attachments are intended only
for the addressee and may contain legally privileged, personal, sensitive or
confidential information. If you are not the intended addressee, and have
received this email, any transmission, distribution, downloading, printing or
photocopying of the contents of this message or attachments is strictly
prohibited. Any legal privilege or confidentiality attached to this message and
attachments is not waived, lost or destroyed by reason of delivery to any
person other than intended addressee. If you have received this message and
are not the intended addressee you should notify the sender by return email and
destroy all copies of the message and any attachments. Unless expressly
attributed, the views expressed in this email do not necessarily represent the
views of the company.
Darryl Bond wrote:> I managed to get it to work something along the lines of your first > method before I read your email. I was also unsuccessful using your > second method.It still eludes me why the second method does not work, but the first seems to be ok too. Users cannot change set setup, so there is no real risk involved. [snip]> I found that if I used VLANs on the Dom0 on eth0 then the trunk would > not pass through to the guests, Hence why Dom0 is not using VLANs on eth0. > I did not do much research to discover why except that the instant I > removed the VLANs from Dom0 eth0 the VLAN networks started working on > the guest.You have to take care here because in Dom0 you have ethX and pethX. Vlans on ethX(dom0) will work whereas those on peth0 apparently do not as expected. My current setup is like this: peth0 -> xenbr0 -> vif0.0 -> eth0(Dom0)+IP-> eth0.42+IP -> vif1.0 -> eth0(Guest1) -> eth0.42+IP -> vif2.0 -> eth0(Guest2) -> eth0.42+IP -> ... Dom0 is part of vlan 42 and my management (no routing between them enabled though). The uplink peth0 is a trunk with management untagged and vlan 42 tagged. My guess why the other setup has problems lies in several parts. Firstly, the management MAC address (default fe:ff:ff:ff:ff:ff) of the second bridge must be different. I forgot to change that and have had problems with that before with STP enabled (bridge can otherwise not be uniquely identified). And secondly, juggling with peth0.X can only be done after xen has remapped the interfaces. The remapping occurs after the original network is brought up. I need to go through the xen scripts to get that right, I guess. I''ll have to do some more experimenting to get this pinned down because it actually is interesting to have bridges that can swap/map/juggle vlan tags. -- Greetings Bertho Bertho Stultiens Senior Systems Manager Mobilethink A/S
On Thu, Nov 23, 2006 at 09:45:20AM +0100, Bertho Stultiens wrote:> Darryl Bond wrote: > > I managed to get it to work something along the lines of your first > > method before I read your email. I was also unsuccessful using your > > second method. > > It still eludes me why the second method does not work, but the first > seems to be ok too. Users cannot change set setup, so there is no real > risk involved. > > > [snip] > > I found that if I used VLANs on the Dom0 on eth0 then the trunk would > > not pass through to the guests, Hence why Dom0 is not using VLANs on eth0. > > I did not do much research to discover why except that the instant I > > removed the VLANs from Dom0 eth0 the VLAN networks started working on > > the guest. > > You have to take care here because in Dom0 you have ethX and pethX. > Vlans on ethX(dom0) will work whereas those on peth0 apparently do not > as expected. My current setup is like this: > > peth0 -> xenbr0 -> vif0.0 -> eth0(Dom0)+IP-> eth0.42+IP > -> vif1.0 -> eth0(Guest1) -> eth0.42+IP > -> vif2.0 -> eth0(Guest2) -> eth0.42+IP > -> ... > > Dom0 is part of vlan 42 and my management (no routing between them > enabled though). The uplink peth0 is a trunk with management untagged > and vlan 42 tagged. > > My guess why the other setup has problems lies in several parts. > Firstly, the management MAC address (default fe:ff:ff:ff:ff:ff) of the > second bridge must be different. I forgot to change that and have had > problems with that before with STP enabled (bridge can otherwise not be > uniquely identified). And secondly, juggling with peth0.X can only be > done after xen has remapped the interfaces. The remapping occurs after > the original network is brought up. I need to go through the xen scripts > to get that right, I guess. > > I''ll have to do some more experimenting to get this pinned down because > it actually is interesting to have bridges that can swap/map/juggle vlan > tags. >Hi! Did you figure out the correct way to do this? (set up bridge per vlan interface in dom0, and export/bridge only untagged frames to domU''s). -- Pasi
Pasi Kärkkäinen wrote:> Did you figure out the correct way to do this? (set up bridge per vlan interface in > dom0, and export/bridge only untagged frames to domU''s).Actually, I never did go further investigating. -- Greetings Bertho Bertho Stultiens Senior Systems Manager Mobilethink A/S
On Mon, Feb 26, 2007 at 09:11:09AM +0100, Bertho Stultiens wrote:> Pasi Kärkkäinen wrote: > > Did you figure out the correct way to do this? (set up bridge per vlan interface in > > dom0, and export/bridge only untagged frames to domU''s). > > Actually, I never did go further investigating. > >OK, thanks for the reply. I''ll see what I can figure out.. -- Pasi