xen discuss - Jun 2009 - Is there a howto for a xen/crossbow vlan setup?

Is there an on-line how to/blog or other showing the steps
for setting up a "network in a box" style virtual LAN
using both Xen paravirtual opensolaris instances and
a combination of crossbow etherstubs/vnics?

I know there are crossbow examples of this using zones,
and I have seen xen references saying you need to
do this using a bridge, but I am unable to find a full
example of this setup on-line.


I am trying to setup an environment like this now, but
I am having a connectivity issue, and I have probably missed
a step or two.

I seem to have both a 2009.06 dom0 and domU with
active vnics on an etherstub, that can ping themselves,
but neither can ping or see the other...
Presumably I have something mis-configured.

A working example from either team would be appreciated.

Thanks in advance,
Doug.

On 06/01/09 11:00, Doug Leavitt wrote:
> Is there an on-line how to/blog or other showing the steps
> for setting up a "network in a box" style virtual LAN
> using both Xen paravirtual opensolaris instances and
> a combination of crossbow etherstubs/vnics?
> 
> I know there are crossbow examples of this using zones,
> and I have seen xen references saying you need to
> do this using a bridge, but I am unable to find a full
> example of this setup on-line.
> 
> 
> I am trying to setup an environment like this now, but
> I am having a connectivity issue, and I have probably missed
> a step or two.
> 
> I seem to have both a 2009.06 dom0 and domU with
> active vnics on an etherstub, that can ping themselves,
> but neither can ping or see the other...
> Presumably I have something mis-configured.
I am confused. You are mentioning etherstubs. They are intended to be 
isolated to the OS instance in which they are installed, as they do not 
need, or use, ''external'' networking hardware. So VNICs on an
etherstub
in one OS instance can not communiate with VNICs on an etherstub in 
another OS instance, regardless of what underlying virtualization you 
are using.

What I don''t know is whether you can extend a VNIC in dom0 to a guest 
domU. I don''t have xVM (or Xen) capable hardware, so I don''t
xVM (the
verb :) )

Steffen
> 
> A working example from either team would be appreciated.
> 
> Thanks in advance,
> Doug.
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

Perhaps I didn''t phrase it correctly.

I have a box installed with os2009.06, as the dom0, running
in hypervisor morde.

I want to install on that box:
multiple domu instances (a mix of both 2008.11, 2009.06,
snv_11x builds etc.) all connected on an internal private
network "in the box".

As I understand it, I need to setup an etherstub for the
''internal ethernet'', the appropriate numbers of vnics,
on the dom0 instance, then as I install specific domu
instances using virt-install  (for instance using a
2008.11 iso) as a paravirtualized opensolaris instance,
I need to specify one of the vnics (say vnic1) as the nic
to used by the domu, so that the domu talks on the
internal network, instead of the default behavior where
the domu tries to use the physical interface for the box.

I believe that the interface to be specified as a bridge
and the script argument needs to be vif-dedicated...
[I think I read that on one of the teams FAQ''s].


e.g.  I am trying to do this:
http://blogs.sun.com/droux/entry/private_virtual_networks_for_solaris
with xVM containers versus zones containers.
[So I can run different OS levels/versions] on the same
internal private {NATd] network.

My problem is my experiments on this has configuration issues, and
I am looking for an example set of virt-install steps [or other]
to explain how to set this type of environment up correctly.

Sort of a combination of droux''s blog and
http://www.opensolaris.org/os/community/xen/docs/opensolaris_domu/
[that is known to actually work].

Thanks in advance,
Doug.






Steffen Weiberle wrote:
> On 06/01/09 11:00, Doug Leavitt wrote:
>> Is there an on-line how to/blog or other showing the steps
>> for setting up a "network in a box" style virtual LAN
>> using both Xen paravirtual opensolaris instances and
>> a combination of crossbow etherstubs/vnics?
>>
>> I know there are crossbow examples of this using zones,
>> and I have seen xen references saying you need to
>> do this using a bridge, but I am unable to find a full
>> example of this setup on-line.
>>
>>
>> I am trying to setup an environment like this now, but
>> I am having a connectivity issue, and I have probably missed
>> a step or two.
>>
>> I seem to have both a 2009.06 dom0 and domU with
>> active vnics on an etherstub, that can ping themselves,
>> but neither can ping or see the other...
>> Presumably I have something mis-configured.
> 
> I am confused. You are mentioning etherstubs. They are intended to be 
> isolated to the OS instance in which they are installed, as they do not 
> need, or use, ''external'' networking hardware. So VNICs on
an etherstub
> in one OS instance can not communiate with VNICs on an etherstub in 
> another OS instance, regardless of what underlying virtualization you 
> are using.
> 
> What I don''t know is whether you can extend a VNIC in dom0 to a
guest
> domU. I don''t have xVM (or Xen) capable hardware, so I
don''t xVM (the
> verb :) )
> 
> Steffen
> 
>>
>> A working example from either team would be appreciated.
>>
>> Thanks in advance,
>> Doug.
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss@opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

Hi Doug,

On 06/01/09 14:12, Doug Leavitt wrote:
> Perhaps I didn''t phrase it correctly.
> 
> I have a box installed with os2009.06, as the dom0, running
> in hypervisor morde.
> 
> I want to install on that box:
> multiple domu instances (a mix of both 2008.11, 2009.06,
> snv_11x builds etc.) all connected on an internal private
> network "in the box".
> 
> As I understand it, I need to setup an etherstub for the
> ''internal ethernet'', the appropriate numbers of vnics,
> on the dom0 instance, then as I install specific domu
> instances using virt-install  (for instance using a
> 2008.11 iso) as a paravirtualized opensolaris instance,
> I need to specify one of the vnics (say vnic1) as the nic
> to used by the domu, so that the domu talks on the
> internal network, instead of the default behavior where
> the domu tries to use the physical interface for the box.
> 
> I believe that the interface to be specified as a bridge
> and the script argument needs to be vif-dedicated...
> [I think I read that on one of the teams FAQ''s].
I think I better understand.
> e.g.  I am trying to do this:
> http://blogs.sun.com/droux/entry/private_virtual_networks_for_solaris
> with xVM containers versus zones containers.
> [So I can run different OS levels/versions] on the same
> internal private {NATd] network.
[Un]fortunately, there are significant differences between using zones 
and xVM, and for the latter, possibly using a physical NIC instead of a 
virtual one. I believe xVM already incorporates VNICs under the covers, 
and a guest on top of a VNIC might be a problem. (I know this is the 
case for LDom--you can''t created a vsw on top of a VNIC, nor a VNIC on 
top of a vnet.)
> 
> My problem is my experiments on this has configuration issues, and
> I am looking for an example set of virt-install steps [or other]
> to explain how to set this type of environment up correctly.
> 
> Sort of a combination of droux''s blog and
> http://www.opensolaris.org/os/community/xen/docs/opensolaris_domu/
> [that is known to actually work].
> 
> Thanks in advance,
> Doug.
Will have to see what the xen-discuss folks say.

Steffen

> Steffen Weiberle wrote:
>> On 06/01/09 11:00, Doug Leavitt wrote:
>>> Is there an on-line how to/blog or other showing the steps
>>> for setting up a "network in a box" style virtual LAN
>>> using both Xen paravirtual opensolaris instances and
>>> a combination of crossbow etherstubs/vnics?
>>>
>>> I know there are crossbow examples of this using zones,
>>> and I have seen xen references saying you need to
>>> do this using a bridge, but I am unable to find a full
>>> example of this setup on-line.
>>>
>>>
>>> I am trying to setup an environment like this now, but
>>> I am having a connectivity issue, and I have probably missed
>>> a step or two.
>>>
>>> I seem to have both a 2009.06 dom0 and domU with
>>> active vnics on an etherstub, that can ping themselves,
>>> but neither can ping or see the other...
>>> Presumably I have something mis-configured.
>>
>> I am confused. You are mentioning etherstubs. They are intended to be 
>> isolated to the OS instance in which they are installed, as they do 
>> not need, or use, ''external'' networking hardware. So
VNICs on an
>> etherstub in one OS instance can not communiate with VNICs on an 
>> etherstub in another OS instance, regardless of what underlying 
>> virtualization you are using.
>>
>> What I don''t know is whether you can extend a VNIC in dom0 to
a guest
>> domU. I don''t have xVM (or Xen) capable hardware, so I
don''t xVM (the
>> verb :) )
>>
>> Steffen
>>
>>>
>>> A working example from either team would be appreciated.
>>>
>>> Thanks in advance,
>>> Doug.
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>

On Mon, Jun 01, 2009 at 01:12:03PM -0500, Doug Leavitt wrote:
> As I understand it, I need to setup an etherstub for the
> ''internal ethernet'', the appropriate numbers of vnics,
> on the dom0 instance, then as I install specific domu
> instances using virt-install  (for instance using a
> 2008.11 iso) as a paravirtualized opensolaris instance,
> I need to specify one of the vnics (say vnic1) as the nic
> to used by the domu, so that the domu talks on the
> internal network, instead of the default behavior where
> the domu tries to use the physical interface for the box.
This is a common mistake - virt-install creates the VNIC for you. So in
fact you want to specify the etherstub as the bridge in the virt-install
line, and it will create a VNIC on top of the stub for you.
> I believe that the interface to be specified as a bridge
> and the script argument needs to be vif-dedicated...
> [I think I read that on one of the teams FAQ''s].
An alternative is to pre-create the VNIC with dladm, then use
vif-dedicated, but there''s currently no way to specify the script via
virt-install, so I don''t recommend that.
> My problem is my experiments on this has configuration issues, and
> I am looking for an example set of virt-install steps [or other]
> to explain how to set this type of environment up correctly.
We''ve been working on:

http://opensolaris.org/os/community/xen/docs/virtinstall/

for precisely this purpose. Note that it assumes 3.3 bits which aren''t
yet putback

regards
john