virt-install has a checkbox which suggests that you can avoid preallocating the entire guest image size -- that the guest image file will grow on demand. Does this work for Xen or is it only supported for KVM and the QEMU COW files? My experience when creating Xen guests as regular files is that I end up with a file that is the full size even when I pick the option to not preallocate the entire image. -- Thanks, Aaron
Aaron Metzger wrote:> virt-install has a checkbox which suggests that you can avoid > preallocating the entire guest image size -- that the guest image file > will grow on demand. > > Does this work for Xen or is it only supported for KVM and the QEMU > COW files?yes, this works fine on xen.> > My experience when creating Xen guests as regular files is that I end > up with a file that is the full size even when I pick the option to > not preallocate the entire image.Basically it if you don''t pre-allocate the entire image, virt-manager creates a sparse file to store the guest data which only occupies disk blocks while you write contents to that sparse file . When checking the size of the pre-allocated and sparse images, you should use "du", not "ls -lh". --Sadique> > > -- > Thanks, > Aaron > > -- > Fedora-xen mailing list > Fedora-xen@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-xen >
Sadique Puthen wrote:> > Basically it if you don''t pre-allocate the entire image, virt-manager > creates a sparse file to store the guest data which only occupies disk > blocks while you write contents to that sparse file . When checking the > size of the pre-allocated and sparse images, you should use "du", not > "ls -lh". >Thank you for your insight. You are correct. I have a guest image stored in a file called "subversion". "du" shows the actual size being used is smaller than what "ls" thinks. du -a subversion 2158768 subversion ls -ld subversion -rwxr-xr-x 1 root root 64424509441 2007-11-27 18:16 subversion I have a follow up question though. How do you cleanly backup and restore these files using the smaller amount of space? Any simplistic program that I try (e.g. "tar") also thinks my file is 60 Gig not 2 Gig and creates a real file that uses 60 Gig of actual disk space. Can anyone share the simple steps with "dd" or other programs that let you cleanly store a backup of the guest image which uses the smaller amount of space and also correctly restores from that backup in a way that preserves the orginal idea of a 60 Gig max guest disk size? I am sorry if these are just basic Xen or even just basic Unix questions that you all already know the answers to as opposed to "fedora-xen" questions but I''m sure I''m not the only one who will run into such things while trying to virtualize their existing infrastructure using Fedora and Xen. This low traffic list has been a wonderful asset in helping with my conversion to Fedora Virtualization. I appreciate all the help I have received. -- Thanks, Aaron
On Sat, Dec 01, 2007 at 12:42:40PM -0500, Aaron Metzger wrote:> Sadique Puthen wrote: > > > > >Basically it if you don''t pre-allocate the entire image, virt-manager > >creates a sparse file to store the guest data which only occupies disk > >blocks while you write contents to that sparse file . When checking the > >size of the pre-allocated and sparse images, you should use "du", not > >"ls -lh". > > > > Thank you for your insight. You are correct. > > I have a guest image stored in a file called "subversion". > > "du" shows the actual size being used is smaller than what "ls" thinks. > > du -a subversion > 2158768 subversion > > ls -ld subversion > -rwxr-xr-x 1 root root 64424509441 2007-11-27 18:16 subversionYou don''t need to use ''du'' - just use the ''-s'' flag to ls $ ls -lsh dan1.img 16K -rwxr-xr-x 1 root root 2.1G 2007-11-29 17:51 dan1.img Shows its physical size is 16k, while its logical size is 2.1 GB.> I have a follow up question though. How do you cleanly backup and > restore these files using the smaller amount of space? Any simplistic > program that I try (e.g. "tar") also thinks my file is 60 Gig not 2 Gig > and creates a real file that uses 60 Gig of actual disk space.Use the --sparse option with tar. -S, --sparse handle sparse files efficiently> Can anyone share the simple steps with "dd" or other programs that let > you cleanly store a backup of the guest image which uses the smaller > amount of space and also correctly restores from that backup in a way > that preserves the orginal idea of a 60 Gig max guest disk size?''cp'' will also try to detect sparse files & handle them properly, but if it gets it wrong you can also use ''--sparse=always'' Dan -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
I think you''ll get the same sort of thing if you gzip the file. I''ve gzip''d some 20G xen images down to around 2G. Daniel P. Berrange wrote:> On Sat, Dec 01, 2007 at 12:42:40PM -0500, Aaron Metzger wrote: > >> Sadique Puthen wrote: >> >> >>> Basically it if you don''t pre-allocate the entire image, virt-manager >>> creates a sparse file to store the guest data which only occupies disk >>> blocks while you write contents to that sparse file . When checking the >>> size of the pre-allocated and sparse images, you should use "du", not >>> "ls -lh". >>> >>> >> Thank you for your insight. You are correct. >> >> I have a guest image stored in a file called "subversion". >> >> "du" shows the actual size being used is smaller than what "ls" thinks. >> >> du -a subversion >> 2158768 subversion >> >> ls -ld subversion >> -rwxr-xr-x 1 root root 64424509441 2007-11-27 18:16 subversion >> > > You don''t need to use ''du'' - just use the ''-s'' flag to ls > > $ ls -lsh dan1.img > 16K -rwxr-xr-x 1 root root 2.1G 2007-11-29 17:51 dan1.img > > Shows its physical size is 16k, while its logical size is 2.1 GB. > > >> I have a follow up question though. How do you cleanly backup and >> restore these files using the smaller amount of space? Any simplistic >> program that I try (e.g. "tar") also thinks my file is 60 Gig not 2 Gig >> and creates a real file that uses 60 Gig of actual disk space. >> > > Use the --sparse option with tar. > > -S, --sparse > handle sparse files efficiently > > >> Can anyone share the simple steps with "dd" or other programs that let >> you cleanly store a backup of the guest image which uses the smaller >> amount of space and also correctly restores from that backup in a way >> that preserves the orginal idea of a 60 Gig max guest disk size? >> > > ''cp'' will also try to detect sparse files & handle them properly, but > if it gets it wrong you can also use ''--sparse=always'' > > Dan >
On Sat, Dec 01, 2007 at 02:03:06PM -0500, Mark Nielsen wrote:> I think you''ll get the same sort of thing if you gzip the file. I''ve > gzip''d some 20G xen images down to around 2G.AFAIR, this is just because sparse files appear as long sequences of zeros & thus compress wel - gzip isn''t actually optimizing for sparseness. The trouble is that when you extract the file gunzip will fully allocate it filling with zeros. My original 1M file takes 4k # ls -lhs foo 4.0K -rw-r--r-- 1 root root 1.0M 2007-12-01 14:18 foo #gzip foo # ls -lhs foo.gz 8.0K -rw-r--r-- 1 root root 1.1K 2007-12-01 14:18 foo.gz Now when it uncompress it it takes up the full 1 MB :-( # gunzip foo.gz # ls -lhs foo 1.1M -rw-r--r-- 1 root root 1.0M 2007-12-01 14:18 foo You *always* want to use tar to preserve sparseness, and then gzip the tar file so you get optimal resource usage both on archiving & extracting. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Aaron Metzger wrote:> virt-install has a checkbox which suggests that you can avoid > preallocating the entire guest image size -- that the guest image file > will grow on demand. > > Does this work for Xen or is it only supported for KVM and the QEMU COW > files?It''s a filesystem property: [root@potoroo ~]# ls -shoog /var/lib/xen/images/* 1.1G -rwxr-xr-x 1 3.1G Nov 7 10:39 /var/lib/xen/images/ClarkConnect.img 2.9G -rwxr-xr-x 1 3.1G Nov 30 18:31 /var/lib/xen/images/DebianEtch.img 4.0G -rwxr-xr-x 1 4.0G Nov 28 17:49 /var/lib/xen/images/F8-1.img 16K -rwxr-xr-x 1 4.0G Nov 28 17:57 /var/lib/xen/images/F8-2.img 16K -rwxr-xr-x 1 4.0G Nov 28 17:38 /var/lib/xen/images/F8.img 16K -rwxr-xr-x 1 4.1G Nov 7 14:06 /var/lib/xen/images/Fedora7.img 16K -rwxr-xr-x 1 2.1G Nov 12 22:55 /var/lib/xen/images/FreeBSD_70.img 129M -rwxr-xr-x 1 4.0G Nov 18 15:51 /var/lib/xen/images/Gentoo.2.img 2.9G -rwxr-xr-x 1 3.1G Nov 18 17:00 /var/lib/xen/images/Gentoo.img 16K -rwxr-xr-x 1 3.1G Nov 5 08:45 /var/lib/xen/images/Kubuntu.img 16K -rwxr-xr-x 1 2.1G Nov 4 10:00 /var/lib/xen/images/NetBSD-31.img 16K -rwxr-xr-x 1 4.0G Nov 28 17:55 /var/lib/xen/images/f8.img 1.1G -rwxr-xr-x 1 2.1G Nov 7 07:52 /var/lib/xen/images/windows_c.img [root@potoroo ~]#> > My experience when creating Xen guests as regular files is that I end up > with a file that is the full size even when I pick the option to not > preallocate the entire image.To interpret my listing above, man ls -- Cheers John -- spambait 1aaaaaaa@coco.merseine.nu Z1aaaaaaa@coco.merseine.nu -- Advice http://webfoot.com/advice/email.top.php http://www.catb.org/~esr/faqs/smart-questions.html http://support.microsoft.com/kb/555375 You cannot reply off-list:-)
Daniel P. Berrange wrote:> On Sat, Dec 01, 2007 at 02:03:06PM -0500, Mark Nielsen wrote: >> I think you''ll get the same sort of thing if you gzip the file. I''ve >> gzip''d some 20G xen images down to around 2G. > > AFAIR, this is just because sparse files appear as long sequences of > zeros & thus compress wel - gzip isn''t actually optimizing for sparseness. > The trouble is that when you extract the file gunzip will fully allocate > it filling with zeros.undo that with "cp --sparse" -- Cheers John -- spambait 1aaaaaaa@coco.merseine.nu Z1aaaaaaa@coco.merseine.nu -- Advice http://webfoot.com/advice/email.top.php http://www.catb.org/~esr/faqs/smart-questions.html http://support.microsoft.com/kb/555375 You cannot reply off-list:-)
Sadique Puthen <sputhenp@redhat.com> writes:> Aaron Metzger wrote: >> virt-install has a checkbox which suggests that you can avoid >> preallocating the entire guest image size -- that the guest image >> file will grow on demand. >> >> Does this work for Xen or is it only supported for KVM and the QEMU >> COW files? > > yes, this works fine on xen.Until you run out of space in the filesystem containing the image file! Then the guest can''t write any blocks that were previously left out of the image file, which leads to massive guest filesystem corruption and data loss. I use sparse image files all the time. But only for the throw-away domains I create for testing. [...]
Markus Armbruster wrote:> Sadique Puthen <sputhenp@redhat.com> writes: > >> Aaron Metzger wrote: >>> virt-install has a checkbox which suggests that you can avoid >>> preallocating the entire guest image size -- that the guest image >>> file will grow on demand. >>> >>> Does this work for Xen or is it only supported for KVM and the QEMU >>> COW files? >> yes, this works fine on xen. > > Until you run out of space in the filesystem containing the image > file! Then the guest can''t write any blocks that were previously left > out of the image file, which leads to massive guest filesystem > corruption and data loss. > > I use sparse image files all the time. But only for the throw-away > domains I create for testing.I thought I read that ext3 journals save the day. Whether they always save it, "man mke2fs" for journal options. Using a device that _is not_ sparse, it seems to me, must be safe. -- Cheers John -- spambait 1aaaaaaa@coco.merseine.nu Z1aaaaaaa@coco.merseine.nu -- Advice http://webfoot.com/advice/email.top.php http://www.catb.org/~esr/faqs/smart-questions.html http://support.microsoft.com/kb/555375 You cannot reply off-list:-)
On Sat, 2007-12-01 at 14:03 -0500, Mark Nielsen wrote:> I think you''ll get the same sort of thing if you gzip the file. I''ve > gzip''d some 20G xen images down to around 2G.Another nice way to compress disk images is with ''qemu-img convert'', outputting to a compressed qcow. David