Xen users - Jun 2007 - Writing a tool for Shared Persistent Windows Boot Image

Before, in my "Hard Problem" email I was trying to communicate a  
design issue were trying to solve with Xen.

This is what we need to do:

1) Deploy 24 Windows XP VMs in parallel.
2) Boot them from a shared Windows XP C: drive.
3) Since this is a read-only shared image we obviously can''t have  
multiple VM''s writing to it.
4) All writes to the boot image for logging, registry and other  
purposes should be diverted to an auxiliary shadow drive specific to  
each VM.
5) After we shut down the VM we need to mount and examine the  
contents of the shadow drive
6) When we are done examining the contents of the shadow drive, we  
need to fast format it for the next VM to use.

Is this supported natively in Xen?  What does everyone else who needs  
to run a lot of Windows VMs do?  There must be a way to support  
shared images.

The reason I posted this to xen-devel is that I could probably  
implement a UnionFS for Windows by writing a kernel hook and  
intercepting all reads and writes to the C: drive, but I don''t have  
enough time to do that right now.  Because of schedule constraints,  
if we don''t find a way to do this in Xen/XenSource we''ll have
to drop
Xen and move on to VMWare ESX at considerable cost to our project.

Are there any senior Xen software engineers out there who''ve done  
this or who might know how?

Thanks,

Jim Burnes


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jim Burnes wrote:
> Before, in my "Hard Problem" email I was trying to communicate a
design
> issue were trying to solve with Xen.
> 
> This is what we need to do:
> 
> 1) Deploy 24 Windows XP VMs in parallel.
I have strong doubts that this would be kosher from a licensing 
perspective, however...
> 2) Boot them from a shared Windows XP C: drive.
> 3) Since this is a read-only shared image we obviously can''t have 
> multiple VM''s writing to it.
> 4) All writes to the boot image for logging, registry and other purposes 
> should be diverted to an auxiliary shadow drive specific to each VM.
If you start with a single image, and then create "COW" files using
the
qcow format, then you can have a shared base image.
> 5) After we shut down the VM we need to mount and examine the contents 
> of the shadow drive
Mounting is tricky.  If you look on qemu-devel, you''ll find a couple 
references to tool that allow you to mount a qcow file (usually with nbd).
> 6) When we are done examining the contents of the shadow drive, we need 
> to fast format it for the next VM to use.
You can just delete the cow file and create a new one.

Regards,

Anthony Liguori
> Is this supported natively in Xen?  What does everyone else who needs to 
> run a lot of Windows VMs do?  There must be a way to support shared images.
> 
> The reason I posted this to xen-devel is that I could probably implement 
> a UnionFS for Windows by writing a kernel hook and intercepting all 
> reads and writes to the C: drive, but I don''t have enough time to
do
> that right now.  Because of schedule constraints, if we don''t find
a way
> to do this in Xen/XenSource we''ll have to drop Xen and move on to
VMWare
> ESX at considerable cost to our project.
> 
> Are there any senior Xen software engineers out there who''ve done
this
> or who might know how?
> 
> Thanks,
> 
> Jim Burnes

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jim Burnes wrote:
>> If you start with a single image, and then create "COW" files
using
>> the qcow format, then you can have a shared base image.
>>
>
> Let me make sure I understand your answer.   COW files are sparse 
> storage for the QEMU environment (which tools you use for Xen).
>
> We want a static filesystem image that represents a snapshot of a 
> Windows XP system right after boot.  When we activate that image we 
> want to perform a few housekeeping issues (like set the MAC and 
> re-DHCP etc), but we also want to make sure that any writes to the 
> image are redirected to an overlay writable file system.  In other 
> words we want a single shared image of the OS itself with all writes 
> going to the COW / shadow image of that specific VM.
COW == Copy On Write.  It''s a separate file that only stores the data 
that has been written since the cow was created.
> Is the COW file you speak of overlayed on top of a single static 
> Windows image or does the COW file contain the entire Windows XP boot 
> image plus any writes.
It just contains the writes after the COW was created.
> I know this depends on your previous answer, but if we delete the COW 
> doesn''t that delete the XP boot image also?
If you delete the COW, the underlying base image is unaffected.
> If the COW file is used as an extent-space to the static Windows XP 
> image, then that should work.
>
> Otherwise I think it would require a recreation of the full Windows XP 
> image.  Ideally that image can be kept static as the continual 
> deletion and re-creation of relatively large COW files would be 
> time-consuming and would tend to fragment hard disk space.
You really shouldn''t worry about disk fragmentation but that''s
a whole
other thread :-)

Regards,

Anthony Liguori
> Other than that, I''m very grateful for your kind assistance.
>
> Thanks again,
>
> Jim Burnes
> Boulder, CO
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Is this supported natively in Xen?  What does everyone else who needs  
> to run a lot of Windows VMs do?  There must be a way to support  
> shared images.
The Linux dom0 kernel has some snapshotting support which seems to be
what you are asking for ?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Alan, thanks for your comment.   Sorry the reply too so long.   I believe
that primitive support for dom0 is available.

So far I''ve been able to implement persistent copy-on-write snapshots
with
LVM snapshotting, however that method seems to consume way to too much main
memory.  We''re limited to about 7 VMs of 256M of memory each. 
I''m trying to
fit about 12 VMs on our 4 GB test box.

To achieve that we''ll probably have to move over to qcow copy-on-write
files.  They seem to be the ideal way to implement our solution, however
I''ve had the devil''s own time getting them to work.  I created
the required
qcow image files and updated my configuration, but ''xm create
<vmname>'' just
runs and the VM immediately quits with no debugging information.

I''m wondering whether (1) I''ve screwed up the qcow images or
(2) I''m using a
set of RPMs that don''t support tap:qcow (using 3.1 RPMS from XenSource)
or
(3) The default RHEL 5 Xen kernel doesn''t include TAP support.

Jim Burnes
Boulder, CO


I''d like to use qcow files, since they seem to do

On 6/21/07, Alan Cox <alan@lxorguk.ukuu.org.uk>
wrote:
>
> > Is this supported natively in Xen?  What does everyone else who needs
> > to run a lot of Windows VMs do?  There must be a way to support
> > shared images.
>
> The Linux dom0 kernel has some snapshotting support which seems to be
> what you are asking for ?
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Anthony (and anyone else listening),

I''ve installed Centos 5, Xen 3.1 (from the Xensource RPMs) and
installed a
base Windows image to an LVM to test.

The installation wen''t fairly well and I resolved various issues like
getting the VMs to render to VNC displays properly.

I even used LVM snapshotting to do some quick copy-on-write tests.  In that
configuration I''ve had up to 7 Windows VMs running at the same time, on
separate IPs, writing to LVM copy-on-write snapshots.  The only problem is
that LVM cow snapshots consume too much main memory.

So I proceeded to my final test which was to replace the LVM storage with
QCOW storage and see how many Windows VMs I could run concurrently. 
(I''m
trying for 12).

The only cow tool I had was qcow-create, so I downloaded the qemu RPMs and
installed them.

I used img2qcow to convert my installed Windows LVM image to the qcow base
image.
I used qcow-img to create a copy-on-write snapshot of the base image.
I tried to boot with this as the new storage type by specifiying:

disk = [ ''tap:qcow:/var/lib/xen/image/winflp.img,hda1,w'',
''file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r'']

(and other variations, by using ''hda'' instead of
''hda1'', using ''tap:aio''
instead of ''tap:qcow'')

But everytime I try to start the vm by ''xm create'' it seems to
create the VM
and the VM immediately exits, leaving hardly any trace of even booting.  I
tried booting directly to hard drive C: and I tried booting to the CDROM.
Booting to the CDROM worked fine but reported no attached hard drive.

The only thing I see is a trace in the xend.log file which indicates that
xend (or the VM) is attempting to generate some sort of hotplug event for
tap.  Then the whole VM shuts down.

Is there something I need to do before tap:qcow works?

I suspect, but am having a hard time proving that either:

(1) My version of Xen 3.1 doesn''t support tap:qcow or
(2) My XenSource kernel doesn''t contain tap support or
(3 I''m loosing my mind

Any ideas that anyone has would be greatly appreciated.

Regards,

Jim Burnes



On 6/21/07, Anthony Liguori <anthony@codemonkey.ws>
wrote:
>
> Jim Burnes wrote:
> >
> > On Jun 21, 2007, at 2:39 PM, Anthony Liguori wrote:
> >
> >> Jim Burnes wrote:
> >>>> If you start with a single image, and then create
"COW" files using
> >>>> the qcow format, then you can have a shared base image.
> >>>>
> >>>
> >>> Let me make sure I understand your answer.   COW files are
sparse
> >>> storage for the QEMU environment (which tools you use for
Xen).
> >>>
> >>> We want a static filesystem image that represents a snapshot
of a
> >>> Windows XP system right after boot.  When we activate that
image we
> >>> want to perform a few housekeeping issues (like set the MAC
and
> >>> re-DHCP etc), but we also want to make sure that any writes to
the
> >>> image are redirected to an overlay writable file system.  In
other
> >>> words we want a single shared image of the OS itself with all
writes
> >>> going to the COW / shadow image of that specific VM.
> >>
> >> COW == Copy On Write.  It''s a separate file that only
stores the data
> >> that has been written since the cow was created.
> >>
> >
> > Excellent.  This is exactly what we need.  I know this is tedious, but
> > just to make absolutely certain (and I''ll check the
documentation),
> > you''re saying that:
> >
> > 1. I can create a COW file and associate it with my C: volume.
>
> Not quite.  You can install Windows to a disk image (drive letters like
> C: are individual partitions within a drive).  You then create a cow
> file that uses that original disk image as a base.
> > 2. All subsequent writes to C: after the COW creation go to the COW
file
>
> All subsequent writes to the drive will go to the COW file after COW
> creation.
>
> > 3. All updates stored in the COW will be read back exactly as if they
> > were on the original C: volume
>
> Yes.
>
> > 4. After VM shutdown, we can attach to the COW image using NBD
> > (network block device) for R/O access
>
> With the appropriate tools, yes.
>
> > 5. We can reset the shared volume back to zero by simply truncating
> > the associated COW file.
>
> Delete and create or:
>
> rm winimg001-cow.img
> qemu-img create -f qcow -b win-base.img winimg001-cow.img
>
> That all happens very quickly.
>
> Regards,
>
> Anthony Liguori
>
> > If this works, it will be great.
> >
> >> You really shouldn''t worry about disk fragmentation but
that''s a
> >> whole other thread :-)
> >>
> >
> > Yeah, I''m always trying to pre-optimize everything 8-)
> >
> > In any case, that can be fixed by using the correct file system.
> >
> > Thanks, Anthony...
> >
> > Jim Burnes
> >
> >
> >
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Jun 28, 2007 at 12:18:47PM -0600, Jim Burnes
wrote:
> Anthony (and anyone else listening),
> 
> I''ve installed Centos 5, Xen 3.1 (from the Xensource RPMs) and
installed a
> base Windows image to an LVM to test.
> 
> The installation wen''t fairly well and I resolved various issues
like
> getting the VMs to render to VNC displays properly.
> 
> I even used LVM snapshotting to do some quick copy-on-write tests.  In that
> configuration I''ve had up to 7 Windows VMs running at the same
time, on
> separate IPs, writing to LVM copy-on-write snapshots.  The only problem is
> that LVM cow snapshots consume too much main memory.
> 
> So I proceeded to my final test which was to replace the LVM storage with
> QCOW storage and see how many Windows VMs I could run concurrently. 
(I''m
> trying for 12).
> 
> The only cow tool I had was qcow-create, so I downloaded the qemu RPMs and
> installed them.
> 
> I used img2qcow to convert my installed Windows LVM image to the qcow base
> image.
> I used qcow-img to create a copy-on-write snapshot of the base image.
> I tried to boot with this as the new storage type by specifiying:
> 
> disk = [ ''tap:qcow:/var/lib/xen/image/winflp.img,hda1,w'',
> ''file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r'']
> 
> (and other variations, by using ''hda'' instead of
''hda1'', using ''tap:aio''
> instead of ''tap:qcow'')
> 
> But everytime I try to start the vm by ''xm create'' it
seems to create the VM
> and the VM immediately exits, leaving hardly any trace of even booting.  I
> tried booting directly to hard drive C: and I tried booting to the CDROM.
> Booting to the CDROM worked fine but reported no attached hard drive.
> 
> The only thing I see is a trace in the xend.log file which indicates that
> xend (or the VM) is attempting to generate some sort of hotplug event for
> tap.  Then the whole VM shuts down.
> 
> Is there something I need to do before tap:qcow works?
> 
> I suspect, but am having a hard time proving that either:
> 
> (1) My version of Xen 3.1 doesn''t support tap:qcow or
It is not possible to use  blktap with HVM guests - there was code which tried
to make it work, but its utterly broken[1]. In my spare time I''m trying
to fix
it, but no ETA.

Dan.
[1] it is calling APIS in XenD which no longer exist & has a try..except
    block which silently catches & ignores the errors, so you never notice
    that it failed.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dan,

Since blktap looks broken on HVM for the forseeable future (a real shame if
you ask me), what''s the next best thing? LVM snapshots?  How can I get
LVM
snapshots to use use less main memory?  Will it help if I decrease the max
size of the copy-on-write snapshot?  Usually our copy-on-write area is no
larger than 20MB.

Jim Burnes


On 6/28/07, Daniel P. Berrange <berrange@redhat.com>
wrote:
>
> On Thu, Jun 28, 2007 at 12:18:47PM -0600, Jim Burnes wrote:
> > Anthony (and anyone else listening),
> >
> > I''ve installed Centos 5, Xen 3.1 (from the Xensource RPMs)
and installed
> a
> > base Windows image to an LVM to test.
> >
> > The installation wen''t fairly well and I resolved various
issues like
> > getting the VMs to render to VNC displays properly.
> >
> > I even used LVM snapshotting to do some quick copy-on-write tests.  In
> that
> > configuration I''ve had up to 7 Windows VMs running at the
same time, on
> > separate IPs, writing to LVM copy-on-write snapshots.  The only
problem
> is
> > that LVM cow snapshots consume too much main memory.
> >
> > So I proceeded to my final test which was to replace the LVM storage
> with
> > QCOW storage and see how many Windows VMs I could run
> concurrently.  (I''m
> > trying for 12).
> >
> > The only cow tool I had was qcow-create, so I downloaded the qemu RPMs
> and
> > installed them.
> >
> > I used img2qcow to convert my installed Windows LVM image to the qcow
> base
> > image.
> > I used qcow-img to create a copy-on-write snapshot of the base image.
> > I tried to boot with this as the new storage type by specifiying:
> >
> > disk = [
''tap:qcow:/var/lib/xen/image/winflp.img,hda1,w'',
> > ''file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r'']
> >
> > (and other variations, by using ''hda'' instead of
''hda1'', using ''tap:aio''
> > instead of ''tap:qcow'')
> >
> > But everytime I try to start the vm by ''xm create''
it seems to create
> the VM
> > and the VM immediately exits, leaving hardly any trace of even
> booting.  I
> > tried booting directly to hard drive C: and I tried booting to the
> CDROM.
> > Booting to the CDROM worked fine but reported no attached hard drive.
> >
> > The only thing I see is a trace in the xend.log file which indicates
> that
> > xend (or the VM) is attempting to generate some sort of hotplug event
> for
> > tap.  Then the whole VM shuts down.
> >
> > Is there something I need to do before tap:qcow works?
> >
> > I suspect, but am having a hard time proving that either:
> >
> > (1) My version of Xen 3.1 doesn''t support tap:qcow or
>
> It is not possible to use  blktap with HVM guests - there was code which
> tried
> to make it work, but its utterly broken[1]. In my spare time I''m
trying to
> fix
> it, but no ETA.
>
> Dan.
> [1] it is calling APIS in XenD which no longer exist & has a
try..except
>     block which silently catches & ignores the errors, so you never
notice
>     that it failed.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
> -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

This should have been fixed a few months ago:

# HG changeset patch
# User wim@xen-wim.site
# Date 1170895981 28800
# Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
# Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
[blktap] Allow HVM booting from blktap device(s)

What problems are you seeing?

a.

On 6/28/07, Daniel P. Berrange <berrange@redhat.com>
wrote:
> On Thu, Jun 28, 2007 at 12:18:47PM -0600, Jim Burnes wrote:
> > Anthony (and anyone else listening),
> >
> > I''ve installed Centos 5, Xen 3.1 (from the Xensource RPMs)
and installed a
> > base Windows image to an LVM to test.
> >
> > The installation wen''t fairly well and I resolved various
issues like
> > getting the VMs to render to VNC displays properly.
> >
> > I even used LVM snapshotting to do some quick copy-on-write tests.  In
that
> > configuration I''ve had up to 7 Windows VMs running at the
same time, on
> > separate IPs, writing to LVM copy-on-write snapshots.  The only
problem is
> > that LVM cow snapshots consume too much main memory.
> >
> > So I proceeded to my final test which was to replace the LVM storage
with
> > QCOW storage and see how many Windows VMs I could run concurrently. 
(I''m
> > trying for 12).
> >
> > The only cow tool I had was qcow-create, so I downloaded the qemu RPMs
and
> > installed them.
> >
> > I used img2qcow to convert my installed Windows LVM image to the qcow
base
> > image.
> > I used qcow-img to create a copy-on-write snapshot of the base image.
> > I tried to boot with this as the new storage type by specifiying:
> >
> > disk = [
''tap:qcow:/var/lib/xen/image/winflp.img,hda1,w'',
> > ''file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r'']
> >
> > (and other variations, by using ''hda'' instead of
''hda1'', using ''tap:aio''
> > instead of ''tap:qcow'')
> >
> > But everytime I try to start the vm by ''xm create''
it seems to create the VM
> > and the VM immediately exits, leaving hardly any trace of even
booting.  I
> > tried booting directly to hard drive C: and I tried booting to the
CDROM.
> > Booting to the CDROM worked fine but reported no attached hard drive.
> >
> > The only thing I see is a trace in the xend.log file which indicates
that
> > xend (or the VM) is attempting to generate some sort of hotplug event
for
> > tap.  Then the whole VM shuts down.
> >
> > Is there something I need to do before tap:qcow works?
> >
> > I suspect, but am having a hard time proving that either:
> >
> > (1) My version of Xen 3.1 doesn''t support tap:qcow or
>
> It is not possible to use  blktap with HVM guests - there was code which
tried
> to make it work, but its utterly broken[1]. In my spare time I''m
trying to fix
> it, but no ETA.
>
> Dan.
> [1] it is calling APIS in XenD which no longer exist & has a
try..except
>     block which silently catches & ignores the errors, so you never
notice
>     that it failed.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
-=|
> |=-           Perl modules: http://search.cpan.org/~danberr/             
-=|
> |=-               Projects: http://freshmeat.net/~danielpb/              
-=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 
-=|
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield
wrote:
> This should have been fixed a few months ago:
> 
> # HG changeset patch
> # User wim@xen-wim.site
> # Date 1170895981 28800
> # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> [blktap] Allow HVM booting from blktap device(s)
> 
> What problems are you seeing?
Yeah I saw that and am still trying to track down why it isn''t doing
what it
ought to. Basically I see the guest start & immediately quit/crash

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

If you can send some more details on the crash we should be able to
sort this out -- it''s certainly something that has worked in the past.

a.

On 6/29/07, Daniel P. Berrange <berrange@redhat.com>
wrote:
> On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield wrote:
> > This should have been fixed a few months ago:
> >
> > # HG changeset patch
> > # User wim@xen-wim.site
> > # Date 1170895981 28800
> > # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> > # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> > [blktap] Allow HVM booting from blktap device(s)
> >
> > What problems are you seeing?
>
> Yeah I saw that and am still trying to track down why it isn''t
doing what it
> ought to. Basically I see the guest start & immediately quit/crash
>
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
-=|
> |=-           Perl modules: http://search.cpan.org/~danberr/             
-=|
> |=-               Projects: http://freshmeat.net/~danielpb/              
-=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 
-=|
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Just as a side note, I''m running from the RHEL 5 3.1.0 RPMs distributed
by
XenSource.  I don''t know if they include this fix.

Regards,

Jim Burnes


On 6/29/07, Andrew Warfield <andrew.warfield@cl.cam.ac.uk>
wrote:
>
> If you can send some more details on the crash we should be able to
> sort this out -- it''s certainly something that has worked in the
past.
>
> a.
>
> On 6/29/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> > On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield wrote:
> > > This should have been fixed a few months ago:
> > >
> > > # HG changeset patch
> > > # User wim@xen-wim.site
> > > # Date 1170895981 28800
> > > # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> > > # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> > > [blktap] Allow HVM booting from blktap device(s)
> > >
> > > What problems are you seeing?
> >
> > Yeah I saw that and am still trying to track down why it
isn''t doing
> what it
> > ought to. Basically I see the guest start & immediately quit/crash
> >
> > Dan.
> > --
> > |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392
> 2496 -=|
> > |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> > |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> > |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
> >
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Speaking of the XenSource 3.1.0 RPMs, I''d like to make a suggestion. 
When
you build the xen-kernel for those 3.1.0 RPMs, can you make sure that the
loop device is built as a module?  By default the loop device is limited to
8 instances and when you build it into the kernel it makes it very difficult
to change max_loop.

I tried adding it as a xen-kernel boot option in grub.conf, but as far as I
can tell it''s ignored when the loop device is compiled in the kernel.

For example, even though I added max_loop=16 to the boot options and
modified the udev configuration to build 16 loop devices, every time xen
does an ''losetup'' on any device greater than loop7 it fails
because the
system doesn''t think the device exists.

Just FYI.  This means I''ll have to raise the limit by recompiling the
whole
kernel.

Jim Burnes


On 6/29/07, Jim Burnes <jvburnes@gmail.com> wrote:
>
> Just as a side note, I''m running from the RHEL 5 3.1.0 RPMs
distributed by
> XenSource.  I don''t know if they include this fix.
>
> Regards,
>
> Jim Burnes
>
>
> On 6/29/07, Andrew Warfield <andrew.warfield@cl.cam.ac.uk> wrote:
> >
> > If you can send some more details on the crash we should be able to
> > sort this out -- it''s certainly something that has worked in
the past.
> >
> > a.
> >
> > On 6/29/07, Daniel P. Berrange < berrange@redhat.com> wrote:
> > > On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield wrote:
> > > > This should have been fixed a few months ago:
> > > >
> > > > # HG changeset patch
> > > > # User wim@xen-wim.site
> > > > # Date 1170895981 28800
> > > > # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> > > > # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> > > > [blktap] Allow HVM booting from blktap device(s)
> > > >
> > > > What problems are you seeing?
> > >
> > > Yeah I saw that and am still trying to track down why it
isn''t doing
> > what it
> > > ought to. Basically I see the guest start & immediately
quit/crash
> > >
> > > Dan.
> > > --
> > > |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978
392
> > 2496 -=|
> > > |=-           Perl modules: http://search.cpan.org/~danberr/
> > <http://search.cpan.org/%7Edanberr/>              -=|
> > > |=-               Projects:
http://freshmeat.net/~danielpb/<http://freshmeat.net/%7Edanielpb/>
> > -=|
> > > |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742
7D3B
> > 9505  -=|
> > >
> >
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield
wrote:
> If you can send some more details on the crash we should be able to
> sort this out -- it''s certainly something that has worked in the
past.
Ok, so here''s the scenario. Traditionally with HVM I have a disk

   file:/xen/rhel4i386.img,hda,w

Having seen the changeset 13827:6524e02edbeb I tried

  tap:aio:/xen/rhel4i386.img,hda,w

And

  tap:aio:/xen/rhel4i386.img,xvda,w

The latter is the preferred, since paravirt drivers should not be hijacking
the IDE devices inside the guest. However, the changeset 13827 doesn''t
seem
to support xvd* since QEMU filters out any devices with such a name.

With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
logging

 qemu: could not open hard disk image ''aio:/xen/rhel4i386.img''

After a little investigation I found that in BlktapController

       try:
           imagetype =
self.vm.info[''image''][''type'']
       except:
           imagetype = ""

Has long ago been broken and should instead be

        try:
            imagetype = self.vm.info.image_type()
        except:
            imagetype = ""


Once I made that change I can see a phantom device being created, but QEMU
still crashes & burns with  qemu-dm-XXXX.log showing

  qemu: could not open hard disk image ''/dev/xvdc1''

I started to debug this, but looking at changeset 13827:6524e02edbeb I rapidly
came to the conclusion that the whole idea of phantom devices is complete
overkill & the entire problem could be addressed with a couple of lines in
ioemu/xenstore.c.  QEMU already knows how to handle all the different 
types of file format blktap does, so there''s no need to setup extra
phantom
devices. All thats needed is for QEMU to a) strip the aio: (or equivalent)
prefix and b) convert xvdN -> hdN if required.

So I''m attaching two patches. The first reverts 13827:6524e02edbeb
and the second tweaks ioemu/xenstore.c so it can handle blktap devices
directly. With these applied ontop of Xen 3.1.0 I can successfully
start HVM guests using  the two example tap:aio lines I show earlier
in this mail. The patch also adds a couple of logging lines which 
end up in qemu-dm-XXX.log as they''ll be useful if ever debugging QEMU
boot issues - it is far too silent when things go wrong which makes
diagnosis hard.

    Signed-off-by: Daniel Berrange <berrange@redhat.com>


The patch being reverted:
 
 $ diffstat xen-revert-phantom.patch 
 ioemu/xenstore.c                           |   46 ---------------------
 python/xen/xend/XendConfig.py              |   41 -------------------
 python/xen/xend/XendDomainInfo.py          |   58 ---------------------------
 python/xen/xend/server/BlktapController.py |   62 -----------------------------
 python/xen/xend/server/DevController.py    |   13 ------
 5 files changed, 3 insertions(+), 217 deletions(-)

The new patch:

 $ diffstat xen-qemu-blktap.patch 
 xenstore.c |   28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel,

Does this take care of both "tap:aio" and "tap:qcow" disk
specifications or
is "tap:qcow" deprecated?

Thanks,

Jim Burnes


On 6/29/07, Daniel P. Berrange <berrange@redhat.com>
wrote:
>
> On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield wrote:
> > If you can send some more details on the crash we should be able to
> > sort this out -- it''s certainly something that has worked in
the past.
>
> Ok, so here''s the scenario. Traditionally with HVM I have a disk
>
>    file:/xen/rhel4i386.img,hda,w
>
> Having seen the changeset 13827:6524e02edbeb I tried
>
>   tap:aio:/xen/rhel4i386.img,hda,w
>
> And
>
>   tap:aio:/xen/rhel4i386.img,xvda,w
>
> The latter is the preferred, since paravirt drivers should not be
> hijacking
> the IDE devices inside the guest. However, the changeset 13827
doesn''t
> seem
> to support xvd* since QEMU filters out any devices with such a name.
>
> With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
> logging
>
> qemu: could not open hard disk image
''aio:/xen/rhel4i386.img''
>
> After a little investigation I found that in BlktapController
>
>        try:
>            imagetype =
self.vm.info[''image''][''type'']
>        except:
>            imagetype = ""
>
> Has long ago been broken and should instead be
>
>         try:
>             imagetype = self.vm.info.image_type()
>         except:
>             imagetype = ""
>
>
> Once I made that change I can see a phantom device being created, but QEMU
> still crashes & burns with  qemu-dm-XXXX.log showing
>
>   qemu: could not open hard disk image ''/dev/xvdc1''
>
> I started to debug this, but looking at changeset 13827:6524e02edbeb I
> rapidly
> came to the conclusion that the whole idea of phantom devices is complete
> overkill & the entire problem could be addressed with a couple of lines
in
> ioemu/xenstore.c.  QEMU already knows how to handle all the different
> types of file format blktap does, so there''s no need to setup
extra
> phantom
> devices. All thats needed is for QEMU to a) strip the aio: (or equivalent)
> prefix and b) convert xvdN -> hdN if required.
>
> So I''m attaching two patches. The first reverts 13827:6524e02edbeb
> and the second tweaks ioemu/xenstore.c so it can handle blktap devices
> directly. With these applied ontop of Xen 3.1.0 I can successfully
> start HVM guests using  the two example tap:aio lines I show earlier
> in this mail. The patch also adds a couple of logging lines which
> end up in qemu-dm-XXX.log as they''ll be useful if ever debugging
QEMU
> boot issues - it is far too silent when things go wrong which makes
> diagnosis hard.
>
>     Signed-off-by: Daniel Berrange <berrange@redhat.com>
>
>
> The patch being reverted:
>
> $ diffstat xen-revert-phantom.patch
> ioemu/xenstore.c                           |   46 ---------------------
> python/xen/xend/XendConfig.py              |   41 -------------------
> python/xen/xend/XendDomainInfo.py          |   58
> ---------------------------
> python/xen/xend/server/BlktapController.py |   62
> -----------------------------
> python/xen/xend/server/DevController.py    |   13 ------
> 5 files changed, 3 insertions(+), 217 deletions(-)
>
> The new patch:
>
> $ diffstat xen-qemu-blktap.patch
> xenstore.c |   28 ++++++++++++++++++++++++++--
> 1 file changed, 26 insertions(+), 2 deletions(-)
>
> Regards,
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
> -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Jun 29, 2007 at 02:32:23PM -0600, Jim Burnes
wrote:
> Daniel,
> 
> Does this take care of both "tap:aio" and "tap:qcow"
disk specifications or
> is "tap:qcow" deprecated?
I''ve not explicitly tested anything other than tap:aio, but QEMU
supports
basically all the formats that blktap does, so I imagine tap:qcow ought
to work with the patch I posted.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 2007-06-29 at 13:07 -0600, Jim Burnes wrote:
> Speaking of the XenSource 3.1.0 RPMs, I''d like to make a
suggestion.
> When you build the xen-kernel for those 3.1.0 RPMs, can you make sure
> that the loop device is built as a module?  By default the loop device
> is limited to 8 instances and when you build it into the kernel it
> makes it very difficult to change max_loop. 
> 
> I tried adding it as a xen-kernel boot option in grub.conf, but as far
> as I can tell it''s ignored when the loop device is compiled in the
> kernel.
When a module is builtin you often need to prefix the option on the
kernel command line with the module name. So adding loop.max_loop=<N>
may work.

Ian.




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The problem with this approach is that you end up using two instances
of whatever virtual disk code you want.  In the case of raw writes to
an image file (tap:aio) this is more or less okay, except for the fact
that qemu has a bad habit of buffering writes and so you can get stuck
in a nasty late write race when you switch from emulated writes over
to using pv drivers.

For non-raw writes this is worse, because most of the image file
drivers cache metadata in memory and don''t necessarily pick up changes
to the image files when you switch from emulated to pv.  Also, the
disparate implementations make me a bit nervous about headers exactly
matching and so on.

The intention of the phantom device is to try to keep things down to a
single data path, and a single driver implementation.

a.

On 6/29/07, Daniel P. Berrange <berrange@redhat.com>
wrote:
> On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield wrote:
> > If you can send some more details on the crash we should be able to
> > sort this out -- it''s certainly something that has worked in
the past.
>
> Ok, so here''s the scenario. Traditionally with HVM I have a disk
>
>    file:/xen/rhel4i386.img,hda,w
>
> Having seen the changeset 13827:6524e02edbeb I tried
>
>   tap:aio:/xen/rhel4i386.img,hda,w
>
> And
>
>   tap:aio:/xen/rhel4i386.img,xvda,w
>
> The latter is the preferred, since paravirt drivers should not be hijacking
> the IDE devices inside the guest. However, the changeset 13827
doesn''t seem
> to support xvd* since QEMU filters out any devices with such a name.
>
> With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
> logging
>
>  qemu: could not open hard disk image
''aio:/xen/rhel4i386.img''
>
> After a little investigation I found that in BlktapController
>
>        try:
>            imagetype =
self.vm.info[''image''][''type'']
>        except:
>            imagetype = ""
>
> Has long ago been broken and should instead be
>
>         try:
>             imagetype = self.vm.info.image_type()
>         except:
>             imagetype = ""
>
>
> Once I made that change I can see a phantom device being created, but QEMU
> still crashes & burns with  qemu-dm-XXXX.log showing
>
>   qemu: could not open hard disk image ''/dev/xvdc1''
>
> I started to debug this, but looking at changeset 13827:6524e02edbeb I
rapidly
> came to the conclusion that the whole idea of phantom devices is complete
> overkill & the entire problem could be addressed with a couple of lines
in
> ioemu/xenstore.c.  QEMU already knows how to handle all the different
> types of file format blktap does, so there''s no need to setup
extra phantom
> devices. All thats needed is for QEMU to a) strip the aio: (or equivalent)
> prefix and b) convert xvdN -> hdN if required.
>
> So I''m attaching two patches. The first reverts 13827:6524e02edbeb
> and the second tweaks ioemu/xenstore.c so it can handle blktap devices
> directly. With these applied ontop of Xen 3.1.0 I can successfully
> start HVM guests using  the two example tap:aio lines I show earlier
> in this mail. The patch also adds a couple of logging lines which
> end up in qemu-dm-XXX.log as they''ll be useful if ever debugging
QEMU
> boot issues - it is far too silent when things go wrong which makes
> diagnosis hard.
>
>     Signed-off-by: Daniel Berrange <berrange@redhat.com>
>
>
> The patch being reverted:
>
>  $ diffstat xen-revert-phantom.patch
>  ioemu/xenstore.c                           |   46 ---------------------
>  python/xen/xend/XendConfig.py              |   41 -------------------
>  python/xen/xend/XendDomainInfo.py          |   58
---------------------------
>  python/xen/xend/server/BlktapController.py |   62
-----------------------------
>  python/xen/xend/server/DevController.py    |   13 ------
>  5 files changed, 3 insertions(+), 217 deletions(-)
>
> The new patch:
>
>  $ diffstat xen-qemu-blktap.patch
>  xenstore.c |   28 ++++++++++++++++++++++++++--
>  1 file changed, 26 insertions(+), 2 deletions(-)
>
> Regards,
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
-=|
> |=-           Perl modules: http://search.cpan.org/~danberr/             
-=|
> |=-               Projects: http://freshmeat.net/~danielpb/              
-=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 
-=|
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, Jul 01, 2007 at 01:28:56PM -0700, Andrew Warfield
wrote:
> The problem with this approach is that you end up using two instances
> of whatever virtual disk code you want.  In the case of raw writes to
> an image file (tap:aio) this is more or less okay, except for the fact
> that qemu has a bad habit of buffering writes and so you can get stuck
> in a nasty late write race when you switch from emulated writes over
> to using pv drivers.
AFAIR, if the guest OS sends a flush request to the IDE device, then
QEMU should immediately be flushing the data to disk in the host - if
it doesn''t, then this is already a potential data corrupter if either
the guest or host crashes because journaling fileystems rely on the
fact that when they ask for a journal flush it is not buffered in RAM.

I don''t think a guest OS would ever be activating both the IDE and
paravirt drivers for a device though would it ? You either load IDE
drivers, or paravirt at any given time. If you''ve got a guest using
PV drivers, then the only point where the IDE interface would come
into play is for the initial BIOS boot process & that should be 
read-only access.
> For non-raw writes this is worse, because most of the image file
> drivers cache metadata in memory and don''t necessarily pick up
changes
> to the image files when you switch from emulated to pv.  Also, the
> disparate implementations make me a bit nervous about headers exactly
> matching and so on.
Are you refering to the qcow file format headers here ? If blktap isn''t
in compliance with the QEMU QCOW spec then that is a bug in itself which
needs fixing, because it is not good for portability of FS images.
> The intention of the phantom device is to try to keep things down to a
> single data path, and a single driver implementation.
Can you explain just how the phantom device is supposed to work, because
its not working in current code, even with the fix I mentioned blow & it
is not immediately clear to me what else needs fixing ...
> >On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield wrote:
> >> If you can send some more details on the crash we should be able
to
> >> sort this out -- it''s certainly something that has worked
in the past.
> >
> >Ok, so here''s the scenario. Traditionally with HVM I have a
disk
> >
> >   file:/xen/rhel4i386.img,hda,w
> >
> >Having seen the changeset 13827:6524e02edbeb I tried
> >
> >  tap:aio:/xen/rhel4i386.img,hda,w
> >
> >And
> >
> >  tap:aio:/xen/rhel4i386.img,xvda,w
> >
> >The latter is the preferred, since paravirt drivers should not be
hijacking
> >the IDE devices inside the guest. However, the changeset 13827
doesn''t seem
> >to support xvd* since QEMU filters out any devices with such a name.
> >
> >With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
> >logging
> >
> > qemu: could not open hard disk image
''aio:/xen/rhel4i386.img''
> >
> >After a little investigation I found that in BlktapController
> >
> >       try:
> >           imagetype =
self.vm.info[''image''][''type'']
> >       except:
> >           imagetype = ""
> >
> >Has long ago been broken and should instead be
> >
> >        try:
> >            imagetype = self.vm.info.image_type()
> >        except:
> >            imagetype = ""
> >
> >
> >Once I made that change I can see a phantom device being created, but
QEMU
> >still crashes & burns with  qemu-dm-XXXX.log showing
> >
> >  qemu: could not open hard disk image ''/dev/xvdc1''
> >
> >I started to debug this, but looking at changeset 13827:6524e02edbeb I 
> >rapidly
> >came to the conclusion that the whole idea of phantom devices is
complete
> >overkill & the entire problem could be addressed with a couple of
lines in
> >ioemu/xenstore.c.  QEMU already knows how to handle all the different
> >types of file format blktap does, so there''s no need to setup
extra phantom
> >devices. All thats needed is for QEMU to a) strip the aio: (or
equivalent)
> >prefix and b) convert xvdN -> hdN if required.
> >
> >So I''m attaching two patches. The first reverts
13827:6524e02edbeb
> >and the second tweaks ioemu/xenstore.c so it can handle blktap devices
> >directly. With these applied ontop of Xen 3.1.0 I can successfully
> >start HVM guests using  the two example tap:aio lines I show earlier
> >in this mail. The patch also adds a couple of logging lines which
> >end up in qemu-dm-XXX.log as they''ll be useful if ever
debugging QEMU
> >boot issues - it is far too silent when things go wrong which makes
> >diagnosis hard.
Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, Jul 01, 2007 at 10:41:37PM +0100, Daniel P. Berrange
wrote:
> On Sun, Jul 01, 2007 at 01:28:56PM -0700, Andrew Warfield wrote:
> > The problem with this approach is that you end up using two instances
> > of whatever virtual disk code you want.  In the case of raw writes to
> > an image file (tap:aio) this is more or less okay, except for the fact
> > that qemu has a bad habit of buffering writes and so you can get stuck
> > in a nasty late write race when you switch from emulated writes over
> > to using pv drivers.
> 
> AFAIR, if the guest OS sends a flush request to the IDE device, then
> QEMU should immediately be flushing the data to disk in the host - if
> it doesn''t, then this is already a potential data corrupter if
either
> the guest or host crashes because journaling fileystems rely on the
> fact that when they ask for a journal flush it is not buffered in RAM.
> 
> I don''t think a guest OS would ever be activating both the IDE and
> paravirt drivers for a device though would it ? You either load IDE
> drivers, or paravirt at any given time. If you''ve got a guest
using
> PV drivers, then the only point where the IDE interface would come
> into play is for the initial BIOS boot process & that should be 
> read-only access.
Thinking about it from the safety POV, the QEMU process could register
a xenstore watch to be notified when the paravirt frontend driver
connected to the backend. At this time it could forceably disable the
IDE device associated with the backend, thus ensuring you never have 
two concurrently active data paths to the same underlying disk.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users