Xen users - Jan 2009 - Can you convert Windows LVM domU to sparse img file?

I have a Windows 2000 domU running in an LVM partition. I need to move
it to another host, but none of my other xen servers have lvm or free
space to create an lvm. So I''d like to convert it to a sparse img file.
The file system in the domU is ntfs. 

Can anyone suggest how to do this?

Thanks,
James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

You can do this by dd.
for example:
dd if=/dev/vg/winpart of=/home/myuser/winimage.img bs=4096
bs set to your block size, as i remember ntfs has 4096 by default.

James Pifer wrote:
> I have a Windows 2000 domU running in an LVM partition. I need to move
> it to another host, but none of my other xen servers have lvm or free
> space to create an lvm. So I''d like to convert it to a sparse img
file.
> The file system in the domU is ntfs. 
>
> Can anyone suggest how to do this?
>
> Thanks,
> James
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>   

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Māris Ruskulis wrote:
> You can do this by dd.
> for example:
> dd if=/dev/vg/winpart of=/home/myuser/winimage.img bs=4096
> bs set to your block size, as i remember ntfs has 4096 by default.
>
bs=32M (or whatever "PE Size" that pvdisplay shows) will probably be
faster.
dd won''t give you SPARSE img file though. Then again, your LV could be
"dirty" (e.g contain deleted data) so that any "normal"
attempt to
create sparse file will not be effective.

I''d suggest you just accept the full,non-sparse file that dd gives you.
An alternative would be to use ntfsresize and ntfsclone, but it requires
a lot more effort (including using kpartx,fdisk, and having a "good"
MBR
handy).

Regards,

Fajar
> James Pifer wrote:
>> I have a Windows 2000 domU running in an LVM partition. I need to move
>> it to another host, but none of my other xen servers have lvm or free
>> space to create an lvm. So I''d like to convert it to a sparse
img file.
>> The file system in the domU is ntfs.
>> Can anyone suggest how to do this?
>>
>> Thanks,
>> James
>>



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Jan 15, 2009, at 3:15 AM, "Fajar A. Nugraha"
<fajar@fajar.net> wrote:
> Māris Ruskulis wrote:
>> You can do this by dd.
>> for example:
>> dd if=/dev/vg/winpart of=/home/myuser/winimage.img bs=4096
>> bs set to your block size, as i remember ntfs has 4096 by default.
>>
> bs=32M (or whatever "PE Size" that pvdisplay shows) will probably
be
> faster.
> dd won''t give you SPARSE img file though. Then again, your LV
could be
> "dirty" (e.g contain deleted data) so that any "normal"
attempt to
> create sparse file will not be effective.
>
> I''d suggest you just accept the full,non-sparse file that dd gives
> you.
> An alternative would be to use ntfsresize and ntfsclone, but it  
> requires
> a lot more effort (including using kpartx,fdisk, and having a
"good"
> MBR
> handy).
1) create a sparse file the size needed

2) add it as a loop device

3) fdisk the loop device

4) ntfsclone the lv to the loop device.

That should do it in theory.

-Ross

>
>> James Pifer wrote:
>>> I have a Windows 2000 domU running in an LVM partition. I need to  
>>> move
>>> it to another host, but none of my other xen servers have lvm or  
>>> free
>>> space to create an lvm. So I''d like to convert it to a
sparse img
>>> file.
>>> The file system in the domU is ntfs.
>>> Can anyone suggest how to do this?
>>>
>>> Thanks,
>>> James
>>>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ross''s solution below is probably the easiest and most practical. 
However, I believe I once saw a utility to create sparse files with DD.  The
first problem with attacking from that angle would be that you would first need
to boot your Windows system and use some utility to zero out all blank space. 
The second problem would be that legitimate 0s might be treated sparsely, and of
course, I could be making this up completely, as such a utility may not even
exist anyway.
	As Māris suggested, the MFT will have to be clean (or you may be able to get
away with a --force in ntfsclone), and you may need kpartx as well, depending on
how your lvm is set up.
	Regarding Ross''s step 3, you can do the same thing with mkfs.ntfs
instead of booting to something for fdisk.  Additionally I''m not
certain you need to do that step at all with ntfsclone anyway, as I believe it
will "reformat" the destination sparse file (by copying the original
MFT) when you run ntfsclone (I suggest this because I used ntfsclone Saturday
and couldn''t go from a 20GB drive to a 10GB one on account of the
destination being too small even though only 6 or 7 GB was in use, and I know
the MFT isn''t 3+ GB, but I don''t remember if I did a --force).
However, if I am correct, it would imply that you also need to keep in mind that
1) it would be necessary to ntfsresize if the destination sparse file was bigger
than the source partition and 2) it would be necessary to ntfsresize before
ntfsclone in order to get to a smaller destination safely (supposing a --force
did work, the thing is, a --force could technically copy to a smaller
destination even if it copied the MFT, and that would certainly upset Windows,
if not other systems, mounting the file after the fact).
	Also, and this is untested theory (in that I don''t know whether or not
you can write directly to a physical/loop device over scp), but you may be able
to mount the loop device on the destination machine and use
scp://<mnt/loop> for the destination file instead of copying the file
after the fact.
	Finally, keep in mind that this sparse file will be just like a MS Virtual PC
vhd file, it will grow as you write and delete.  In order to bring it back down,
you would have to do something like my first paragraph suggests, or follow
Ross''s steps again from one loop device to another.  This is why I
think I recall seeing a utility to do that, though, I believe a sparse file
mounted loop with ext3 on it would have the same problem growing with writes and
deletes otherwise, though not necessarily to the same degree, and it seems like
I saw such a solution for that a long time ago.
	Dustin

-----Original Message-----
From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Ross Walker
Sent: Thursday, January 15, 2009 09:25
To: Fajar A. Nugraha
Cc: Xen Users Mailing List
Subject: Re: [Xen-users] Can you convert Windows LVM domU to sparse img file?

On Jan 15, 2009, at 3:15 AM, "Fajar A. Nugraha"
<fajar@fajar.net> wrote:
> Māris Ruskulis wrote:
>> You can do this by dd.
>> for example:
>> dd if=/dev/vg/winpart of=/home/myuser/winimage.img bs=4096
>> bs set to your block size, as i remember ntfs has 4096 by default.
>>
> bs=32M (or whatever "PE Size" that pvdisplay shows) will probably
be
> faster.
> dd won''t give you SPARSE img file though. Then again, your LV
could be
> "dirty" (e.g contain deleted data) so that any "normal"
attempt to
> create sparse file will not be effective.
>
> I''d suggest you just accept the full,non-sparse file that dd gives
> you.
> An alternative would be to use ntfsresize and ntfsclone, but it  
> requires
> a lot more effort (including using kpartx,fdisk, and having a
"good"
> MBR
> handy).
1) create a sparse file the size needed

2) add it as a loop device

3) fdisk the loop device

4) ntfsclone the lv to the loop device.

That should do it in theory.

-Ross

>
>> James Pifer wrote:
>>> I have a Windows 2000 domU running in an LVM partition. I need to  
>>> move
>>> it to another host, but none of my other xen servers have lvm or  
>>> free
>>> space to create an lvm. So I''d like to convert it to a
sparse img
>>> file.
>>> The file system in the domU is ntfs.
>>> Can anyone suggest how to do this?
>>>
>>> Thanks,
>>> James
>>>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I was able to do it using Māris'' suggestion using:
dd if=/dev/vg/winpart of=/home/myuser/winimage.img bs=4096

I made sure the system was shut down cleanly and successfully before
doing it. Once it was done I used ''cp -v --sparse=always...''
and copied
it to another file. 

Using ''du -s -B1 filename'' I confirmed the file size. 

I recreated the machine in virtmanager and it booted up and seems to be
good. 

Thanks for the suggestions. 

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ross Walker wrote:
> On Jan 15, 2009, at 3:15 AM, "Fajar A. Nugraha"
<fajar@fajar.net> wrote:
>> An alternative would be to use ntfsresize and ntfsclone, but it
requires
>> a lot more effort (including using kpartx,fdisk, and having a
"good" MBR
>> handy).
>
I wrote "a lot more effort" because it does, indeed, require a lot
more
effort. I did something similar actually (cloning Windows on an LV to
another LV).
> 1) create a sparse file the size needed
>
> 2) add it as a loop device
>
> 3) fdisk the loop device
>
During fdisk, there are some caveats :
- make SURE partition 1 (or whatever your windows partition will be) is
set active
- set the type to 7 (NTFS)

In my case I also had make sure that the last cylinder is unused.
For example, if you create 10G LV (or disk image), and fdisk shows 1035
cylinders, partition 1 can only occupy cylinder 1-1034 (This is what you
get if you install windows from CD on that LV). Linux''s fdisk let you
use cylinder 1-1035, but if you do this Windows will BSOD. Took me
several hours to figure this out by comparing original and cloned machine.

After this, you need two more steps :
- get a good MBR, copy it with dd. I used the one from syslinux
(/usr/share/syslinux/mbr.bin)
- create dev mappings for the partition using kpartx (something like
kpartx -av /dev/loop0). It will be in /dev/mapper/loop0p1 (or something
similar). Do this for both the LV and the loop device.
> 4) ntfsclone the lv to the loop device.
>
Regards,

Fajar


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

James Pifer wrote:
> I was able to do it using Māris'' suggestion using:
> dd if=/dev/vg/winpart of=/home/myuser/winimage.img bs=4096
>
> I made sure the system was shut down cleanly and successfully before
> doing it. Once it was done I used ''cp -v
--sparse=always...'' and copied
> it to another file. 
>
> Using ''du -s -B1 filename'' I confirmed the file size. 
>
> I recreated the machine in virtmanager and it booted up and seems to be
> good. 
>
>   
Good to know then.
Just to add an info, if the unused space on your windows does not
contain zeroes (might happen as a result of file deletion, or running
defrag) then your sparse file might not be 100% effective (as in you
should be able to reduce the size further).

VirtualBox''s help page suggests running Microsoft''s sdelete
tool to
zero-out unsused space.
A similar result can also be achieved using ntfsclone and ntfsresize :
- ntfsresize the original disk to the minimum size
- copy using ntfsclone
- ntfsresize the clone to occupy full space

The sdelete method is easier to use, while the ntfsclone/resize method
has the benefit of being space-efficient (e.g. you don''t need to have a
temporary non-sparse image file first).

Regards,

Fajar



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Wow, I seriously misread the e-mails when I responded yesterday, thinking
"format" in place of "fdisk" and "MFT" in place of
"MBR."  The sad thing is, I just did this a week ago, and had to go
back and mark the partition active because I had forgotten and couldn''t
boot, yet when responding, thinking it was still all fresh in my mind, it
wasn''t.  That said, James, congrats on doing the sparse file the way
you wanted to.  Anyone else following this thread, Fajar''s information
below is as or more important than the information I provided yesterday. 
Finally, Fajar, you should be able to use the final cylinder without Windows
BSODing, I do it all of the time, sometimes by partitioning prior to
installation and other times by cloning.  I am guessing the problem causing your
BSOD was that the NTFS MFT indicated a different size than the partition.  In
that case, an ntfsresize would have fixed it as well, not that the extra 8MB
matters.
	Dustin

-----Original Message-----
From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Fajar A. Nugraha
Sent: Thursday, January 15, 2009 22:00
To: Xen Users Mailing List
Subject: Re: [Xen-users] Can you convert Windows LVM domU to sparse img file?

Ross Walker wrote:
> On Jan 15, 2009, at 3:15 AM, "Fajar A. Nugraha"
<fajar@fajar.net> wrote:
>> An alternative would be to use ntfsresize and ntfsclone, but it 
>> requires a lot more effort (including using kpartx,fdisk, and having 
>> a "good" MBR handy).
>
I wrote "a lot more effort" because it does, indeed, require a lot
more effort. I did something similar actually (cloning Windows on an LV to
another LV).
> 1) create a sparse file the size needed
>
> 2) add it as a loop device
>
> 3) fdisk the loop device
>
During fdisk, there are some caveats :
- make SURE partition 1 (or whatever your windows partition will be) is set
active
- set the type to 7 (NTFS)

In my case I also had make sure that the last cylinder is unused.
For example, if you create 10G LV (or disk image), and fdisk shows 1035
cylinders, partition 1 can only occupy cylinder 1-1034 (This is what you get if
you install windows from CD on that LV). Linux''s fdisk let you use
cylinder 1-1035, but if you do this Windows will BSOD. Took me several hours to
figure this out by comparing original and cloned machine.

After this, you need two more steps :
- get a good MBR, copy it with dd. I used the one from syslinux
(/usr/share/syslinux/mbr.bin)
- create dev mappings for the partition using kpartx (something like kpartx -av
/dev/loop0). It will be in /dev/mapper/loop0p1 (or something similar). Do this
for both the LV and the loop device.
> 4) ntfsclone the lv to the loop device.
>
Regards,

Fajar



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users