xen discuss - Oct 2009 - Status of xVM in OpenSolaris for production (cloud)

Hello,

I''m building a "Sun Cloud" based on Sun hardware and software
for
hosting.

The storage will be, of course, a Sun Storage 7000 iSCSI (2xGbE  
MPxIO), the
hosts servers will be Sun Fire X4170 with either :
    - Linux/Debian + XenSource + OpenVSwitch (citrix project)
    - or OpenSolaris + xVM + crossbow

I prefer the xVM solution because it''s integrated to OpenSolaris, uses
libvirt
and supports out of the box network virtualization.

The guests (domU) will be only paravirtualized Debian with custom  
kernel,
and maybe some OpenSolaris x86pv (I will use zones for Solaris/ 
OpenSolaris
guests).

The features that are mandatory are :
    - PV guest support (local kernel, no pygrub)
    - Live migration
    - Virtualized network (bonding, spanning-tree, vlans, bw limiting,  
accounting)
    - API for remote management (libvirt is ok)
    - stability and performance (200+ managed guests on 5 hosts)

My tests are done now with OpenSolaris 2010.02-snv_124 (Xen 3.3.2), it  
works
well but I have not tested the network and performances.


So, is xVM (snv_124) stable enough for production ?


When will xvm-3.4 come to OpenSolaris ?


Why is Sun using virtualbox for his cloud ?


Regards,


Frédéric.

>The guests (domU) will be only paravirtualized Debian with custom kernel,
>and maybe some OpenSolaris x86pv (I will use zones for Solaris/OpenSolaris
>guests).
If you like libvirt so much, i.e. virt-install as core tool for PV DomU
installation and
pygrub to load PV DomU in post-installation phase, then how you intend to
install paravirtualized Debian with custom kernel without debootstrap integrated
into system. Any Ubuntu PV guest will be affected in same way. Ubuntu
8.10,9.04,9.10 Servers ISO images are completely useless for virt-install,
rather then to create HVM DomUs and try to utilize image devices for  PV guest
load, what is not approved method for Debian PV guests install vs debootstrap.

View for instance :-
http://blog.adventuresinopensolaris.com/2008/07/xvm-pv-of-debian-distros.html


--- On Fri, 10/9/09, Frédéric VANNIÈRE <frederic@vanniere.fr> wrote:

From: Frédéric VANNIÈRE <frederic@vanniere.fr>
Subject: [xen-discuss] Status of xVM in OpenSolaris for production (cloud)
To: "Xen discuss" <xen-discuss@opensolaris.org>
Date: Friday, October 9, 2009, 4:01 AM

Hello,

I''m building a "Sun Cloud" based on Sun hardware and software
for hosting.

The storage will be, of course, a Sun Storage 7000 iSCSI (2xGbE MPxIO), the
hosts servers will be Sun Fire X4170 with either :
   - Linux/Debian + XenSource + OpenVSwitch (citrix project)
   - or OpenSolaris + xVM + crossbow

I prefer the xVM solution because it''s integrated to OpenSolaris, uses
libvirt
and supports out of the box network virtualization.

The guests (domU) will be only paravirtualized Debian with custom kernel,
and maybe some OpenSolaris x86pv (I will use zones for Solaris/OpenSolaris
guests).

The features that are mandatory are :
   - PV guest support (local kernel, no pygrub)
   - Live migration
   - Virtualized network (bonding, spanning-tree, vlans, bw limiting,
accounting)
   - API for remote management (libvirt is ok)
   - stability and performance (200+ managed guests on 5 hosts)

My tests are done now with OpenSolaris 2010.02-snv_124 (Xen 3.3.2), it works
well but I have not tested the network and performances.


So, is xVM (snv_124) stable enough for production ?


When will xvm-3.4 come to OpenSolaris ?


Why is Sun using virtualbox for his cloud ?


Regards,


Frédéric.
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

Le 9 oct. 2009 à 11:00, Boris Derzhavets a écrit :
> >The guests (domU) will be only paravirtualized Debian with custom  
> kernel,
> >and maybe some OpenSolaris x86pv (I will use zones for Solaris/ 
> OpenSolaris
> >guests).
>
> If you like libvirt so much, i.e. virt-install as core tool for PV  
> DomU installation and
> pygrub to load PV DomU in post-installation phase, then how you  
> intend to install paravirtualized Debian with custom kernel without  
> debootstrap integrated into system. Any Ubuntu PV guest will be  
> affected in same way. Ubuntu 8.10,9.04,9.10 Servers ISO images are  
> completely useless for virt-install, rather then to create HVM DomUs  
> and try to utilize image devices for  PV guest load, what is not  
> approved method for Debian PV guests install vs debootstrap.
>
> View for instance :-
>
http://blog.adventuresinopensolaris.com/2008/07/xvm-pv-of-debian-distros.html

I use the same process for real servers and VMs. The server boots on a  
custom nfsroot and runs automatically the install script. I don''t use  
virt-install but I use virsh with a custom XML description of the  
server.

    1. create the XML description of the VM
    2. load it with "virsh loadxml ..."
    3. boot the VM (nfsroot + install)
    4. modify the XML description so the VM boots on its disk
    5. reboot the VM.

That works well with Debian.

>1. create the XML description of the VM
>2. load it with "virsh loadxml ..."
I was unable find option "loadxml"  for virsh , just
"create".
I believe, that start-up xml file should include lines kind of 
<kernel>vmlinuz-install-xen</kernel>
<ramdisk>intird-install-xen</ramdisk>
for particular Debian DomU.
Not sure would it work for pvops kernels (Ubuntu > 8.10)
Could you share just one installation sample for Debian DomU ?
-- 
This message posted from opensolaris.org

To be honest, my concern is straight forward install Ubuntu 9.04 (8.10) PV DomUs
at Xen 3.4.1 F11 Dom0. Xen Host been installed on F11 via xen-3.4.1-4.src.rpm
plus Libvirt 0.7.1-10 does provide completely functional virt-manager (
virt-install).  I don''t like either tricks to install Ubuntu PV or
debootstrapping ( F11 does have debootstrap.f11.noarch.rpm)
-- 
This message posted from opensolaris.org

> So, is xVM (snv_124) stable enough for production ?
>
>
> When will xvm-3.4 come to OpenSolaris ?
Is this in the 2010 release currently for testing?

* jsacksteder@gmail.com [2009-10-09 14:03:58]
>> So, is xVM (snv_124) stable enough for production ?
>>
>>
>> When will xvm-3.4 come to OpenSolaris ?
>
> Is this in the 2010 release currently for testing?
No.

dme.
-- 
David Edmondson, Sun Microsystems, http://dme.org

Boris Derzhavets wrote:
>  >The guests (domU) will be only paravirtualized Debian with custom
kernel,
>  >and maybe some OpenSolaris x86pv (I will use zones for
Solaris/OpenSolaris
>  >guests).
> 
> If you like libvirt so much, i.e. virt-install as core tool for PV DomU 
> installation and
> pygrub to load PV DomU in post-installation phase, then how you intend 
> to install paravirtualized Debian with custom kernel without debootstrap 
> integrated into system. Any Ubuntu PV guest will be affected in same 
> way. Ubuntu 8.10,9.04,9.10 Servers ISO images are completely useless for 
> virt-install, rather then to create HVM DomUs and try to utilize image 
> devices for  PV guest load, what is not approved method for Debian PV 
> guests install vs debootstrap.
> 
> View for instance :-
>
http://blog.adventuresinopensolaris.com/2008/07/xvm-pv-of-debian-distros.html
> 
The following works fine for me...

    virt-install -n debian -r 1024  -p --nographics -f
/dev/zvol/dsk/rpool/xvm/debian  -l
http://ftp.us.debian.org/debian/dists/lenny/main/installer-i386/


MRJ

Frédéric VANNIÈRE wrote:
> Hello,
> 
> I''m building a "Sun Cloud" based on Sun hardware and
software for hosting.
> 
> The storage will be, of course, a Sun Storage 7000 iSCSI (2xGbE MPxIO), the
> hosts servers will be Sun Fire X4170 with either :
>    - Linux/Debian + XenSource + OpenVSwitch (citrix project)
>    - or OpenSolaris + xVM + crossbow
> 
> I prefer the xVM solution because it''s integrated to OpenSolaris,
uses
> libvirt
> and supports out of the box network virtualization.
> 
> The guests (domU) will be only paravirtualized Debian with custom kernel,
> and maybe some OpenSolaris x86pv (I will use zones for Solaris/OpenSolaris
> guests).
> 
> The features that are mandatory are :
>    - PV guest support (local kernel, no pygrub)
local kernel (on dom0) won''t be great for Solaris..


>    - Live migration
>    - Virtualized network (bonding, spanning-tree, vlans, bw limiting, 
> accounting)
>    - API for remote management (libvirt is ok)
>    - stability and performance (200+ managed guests on 5 hosts)
> 
> My tests are done now with OpenSolaris 2010.02-snv_124 (Xen 3.3.2), it 
> works
> well but I have not tested the network and performances.
> 
> 
> So, is xVM (snv_124) stable enough for production ?
It''s a development build. You will have to make your own determination
there... I have been happy with it, but I''m not using it in production
:-)



> 
> When will xvm-3.4 come to OpenSolaris ?
3.4 is in testing... When we get through the qual, it will go back...
There are 4 or 5 stopper bugs right now with more testing left to
complete.

We broke this one into multiple putbacks.. We''ve pushed updates to
the vdisk code and libvirt code already. Only the xen portion is left.



> 
> Why is Sun using virtualbox for his cloud ?
Reference?  I don''t know what it will or will not be using...

VirtualBox works really nice for environments where a graphics head
is important. But you will get a decent hit on I/O performance
and SMP performance.

I wouldn''t be surprised to see a mixture of both depending on what
problem your trying to solve...  But I don''t have any knowledge either
way...





MRJ

FWIW,

We''ve been running a small "cloud" of about 20 servers
running xVM Xen.
Initially Nevada hypervisors but now running OpenSolaris 2009.06. Each 
hypervisor runs typically between 5-15 guests (mostly OpenSolaris 
guests). All the guests virtual disks are hosted on a 7410 Storage 
Cluster and using iSCSI.

For some details: http://blogs.sun.com/mock/entry/my_own_private_cloud_aho1

In general its all been pretty stable for us. It all works quite well. 
Gives us reasonable performance and stability.

The initial iSCSI support on the Unified Storage had some limitations 
though, this is where the biggest head aches were. The latest firmware 
is supposed to be much more robust, which we happen to be migrating to 
today. My recommendation is if you plan to cluster your 7000 boxes, make 
sure you test failover and takeback and force faults (such as head 
reboots, network outages, etc) so you understand and have an expectation 
on how everything will behave before the condition occurs in a 
production environment.

  --joe

Mark Johnson wrote:
>
>
> Frédéric VANNIÈRE wrote:
>> Hello,
>>
>> I''m building a "Sun Cloud" based on Sun hardware and
software for
>> hosting.
>>
>> The storage will be, of course, a Sun Storage 7000 iSCSI (2xGbE 
>> MPxIO), the
>> hosts servers will be Sun Fire X4170 with either :
>>    - Linux/Debian + XenSource + OpenVSwitch (citrix project)
>>    - or OpenSolaris + xVM + crossbow
>>
>> I prefer the xVM solution because it''s integrated to
OpenSolaris,
>> uses libvirt
>> and supports out of the box network virtualization.
>>
>> The guests (domU) will be only paravirtualized Debian with custom 
>> kernel,
>> and maybe some OpenSolaris x86pv (I will use zones for 
>> Solaris/OpenSolaris
>> guests).
>>
>> The features that are mandatory are :
>>    - PV guest support (local kernel, no pygrub)
>
> local kernel (on dom0) won''t be great for Solaris..
>
>
>
>>    - Live migration
>>    - Virtualized network (bonding, spanning-tree, vlans, bw limiting, 
>> accounting)
>>    - API for remote management (libvirt is ok)
>>    - stability and performance (200+ managed guests on 5 hosts)
>>
>> My tests are done now with OpenSolaris 2010.02-snv_124 (Xen 3.3.2), 
>> it works
>> well but I have not tested the network and performances.
>>
>>
>> So, is xVM (snv_124) stable enough for production ?
>
> It''s a development build. You will have to make your own
determination
> there... I have been happy with it, but I''m not using it in
production
> :-)
>
>
>
>
>>
>> When will xvm-3.4 come to OpenSolaris ?
>
> 3.4 is in testing... When we get through the qual, it will go back...
> There are 4 or 5 stopper bugs right now with more testing left to
> complete.
>
> We broke this one into multiple putbacks.. We''ve pushed updates to
> the vdisk code and libvirt code already. Only the xen portion is left.
>
>
>
>
>>
>> Why is Sun using virtualbox for his cloud ?
>
> Reference?  I don''t know what it will or will not be using...
>
> VirtualBox works really nice for environments where a graphics head
> is important. But you will get a decent hit on I/O performance
> and SMP performance.
>
> I wouldn''t be surprised to see a mixture of both depending on what
> problem your trying to solve...  But I don''t have any knowledge
either
> way...
>
>
>
>
>
> MRJ
>
>
> _______________________________________________
> xen-discuss mailing list
> xen-discuss@opensolaris.org

> Could you share just one installation sample for
> Debian DomU ?
That''s virsh start f12.xml for fedora, sorry for stupid question:-

<domain type=''xen''>
  <name>VMFZ</name>
<os>
  <type>linux</type>
 
<kernel>/export/home/boris/fedora12/vmlinuz</kernel>
 
<initrd>/export/home/boris/fedora12/initrd.img</initrd>
</os>
  <memory>1048576</memory>
  <vcpu>1</vcpu>
  <devices>
    <disk type=''block''
device=''disk''>
      <driver name=''phy''
type=''zvol''/>
      <source dev=''/tank03/disk-linux''/>
      <target dev=''xvda''
bus=''xen''/>
    </disk>
    <interface type=''bridge''>
      <source bridge=''rge0''/>
      <mac address=''00:16:36:43:2a:72''/>
    </interface>
    <input type=''mouse''
bus=''xen''/>
    <graphics type=''vnc'' port=''-1''
keymap=''en-us''/>
  </devices>
</domain>
-- 
This message posted from opensolaris.org

> The following works fine for me...
> 
> virt-install -n debian -r 1024  -p --nographics
> -f /dev/zvol/dsk/rpool/xvm/debian  -l
>  http://ftp.us.debian.org/debian/dists/lenny/main/inst
> ller-i386/
> 
> 
> MRJ
Thank you, Mark.
I guess it works , because installation initrd.img ( or whatever)  contains xen
frontend drivers for DomU
-- 
This message posted from opensolaris.org

--location \
http://ftp.us.debian.org/debian/dists/lenny/main/installer-amd64

immediately fails with message "unable to retrieve xen kernel" 



--- On Fri, 10/9/09, Mark Johnson <Mark.Johnson@Sun.COM> wrote:

From: Mark Johnson <Mark.Johnson@Sun.COM>
Subject: Re: [xen-discuss] Status of xVM in OpenSolaris for production (cloud)
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "Xen discuss" <xen-discuss@opensolaris.org>, "Frédéric
VANNIÈRE" <frederic@vanniere.fr>
Date: Friday, October 9, 2009, 1:09 PM



Boris Derzhavets wrote:
>  >The guests (domU) will be only paravirtualized Debian with custom
kernel,
>  >and maybe some OpenSolaris x86pv (I will use zones for
Solaris/OpenSolaris
>  >guests).
> 
> If you like libvirt so much, i.e. virt-install as core tool for PV DomU
installation and
> pygrub to load PV DomU in post-installation phase, then how you intend to
install paravirtualized Debian with custom kernel without debootstrap integrated
into system. Any Ubuntu PV guest will be affected in same way. Ubuntu
8.10,9.04,9.10 Servers ISO images are completely useless for virt-install,
rather then to create HVM DomUs and try to utilize image devices for  PV guest
load, what is not approved method for Debian PV guests install vs debootstrap.
> 
> View for instance :-
>
http://blog.adventuresinopensolaris.com/2008/07/xvm-pv-of-debian-distros.html
> 
The following works fine for me...

   virt-install -n debian -r 1024  -p --nographics -f
/dev/zvol/dsk/rpool/xvm/debian  -l
http://ftp.us.debian.org/debian/dists/lenny/main/installer-i386/


MRJ

On Sat, Oct 10, 2009 at 01:47:37AM -0700, Boris Derzhavets wrote:
> --location \
> http://ftp.us.debian.org/debian/dists/lenny/main/installer-amd64
> 
> immediately fails with message "unable to retrieve xen kernel" 
Correct, Debian do not provide a 64-bit Xen kernel.

regards
john

root@opensolaris:/export/home/boris/f12# cat start.xml
<domain type=''xen''>
<name>VMF12</name>
<os>
<type>linux</type>
<kernel>/export/home/boris/f12/vmlinuz</kernel>
<initrd>/export/home/boris/f12/initrd.img</initrd>
</os>
<memory>1048576</memory>
<vcpu>1</vcpu>
<devices>
<disk type=''block'' device=''disk''>
<driver name=''phy'' type=''zvol''/>
<source dev=''/tank03/disk''/>
<target dev=''xvda'' bus=''xen''/>
</disk>
<interface type=''bridge''>
<source bridge=''rge0''/>
<mac address=''00:16:36:43:2a:72''/>
</interface>
<input type=''mouse'' bus=''xen''/>
<graphics type=''vnc'' port=''-1''
keymap=''en-us''/>
</devices>
</domain>
# virsh create start.xml
Retrieving install.img hangs at 63%, 53%,8% depending on your luck.

Virt-install refuse --os-variant fedora12-beta immediatelly

Finally, i was able to run virt-install via NFS shared ISO image.
Anaconda is moving pretty slow vs  performance at  Xen 3.4.1 Dom0 F11
on the same box. ( /boot was changed to ext3fs)

--- On Wed, 10/21/09, Boris Derzhavets <bderzhavets@yahoo.com> wrote:

From: Boris Derzhavets <bderzhavets@yahoo.com>
Subject: [xen-discuss] Attempt to install F12 beta PV DomU at OSOL-2010-02-124
Dom0 via local HTTP mirror
To: xen-discuss@opensolaris.org
Date: Wednesday, October 21, 2009, 7:43 AM

root@opensolaris:/export/home/boris/f12# cat start.xml
<domain type=''xen''>
<name>VMF12</name>
<os>
<type>linux</type>
<kernel>/export/home/boris/f12/vmlinuz</kernel>
<initrd>/export/home/boris/f12/initrd.img</initrd>
</os>
<memory>1048576</memory>
<vcpu>1</vcpu>
<devices>
<disk type=''block'' device=''disk''>
<driver name=''phy'' type=''zvol''/>
<source dev=''/tank03/disk''/>
<target dev=''xvda'' bus=''xen''/>
</disk>
<interface type=''bridge''>
<source bridge=''rge0''/>
<mac address=''00:16:36:43:2a:72''/>
</interface>
<input type=''mouse'' bus=''xen''/>
<graphics type=''vnc'' port=''-1''
keymap=''en-us''/>
</devices>
</domain>
# virsh create
 start.xml
Retrieving install.img hangs at 63%, 53%,8% depending on your luck.

Virt-install refuse --os-variant fedora12-beta immediatelly






      
-----Inline Attachment Follows-----

_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org