libvirt_users@skagitattic.com
2016-Jun-15 15:20 UTC
[libvirt-users] virt-sparsify changing the apparent-size of files
Hello, I am curious why when using virt-sparsify the apparent-size of the file gets reduced to the actual file size? Is there a way to get the apparent-size left as the full size? In the man page for virt-sparsify it has a section on "IMPORTANT NOTE ABOUT SPARSE OUTPUT IMAGES" and its output seems to indicate what I would expect where "ls -lh" shows the full apparent size and "du -sh" shows the actual size. But my actual results from testing are different. In the following example input.qcow2 is before sparsify and output.qcow2 is after. Doing ls reports full size for input but a small size for the output: root@testingbox:/tmp/test # ls -lh total 3.1G -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img -rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2 -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2 Same when asking du for apparent-size: root@testingbox:/tmp/test # du -h --apparent-size * 512M file.img 51G input.qcow2 790M output.qcow2 A normal du will show the actual size of the files: root@testingbox:/tmp/test # du -h * 0 file.img 2.4G input.qcow2 790M output.qcow2 The only way I have found to check the full virtual size of the file after virt-sparsify is with qemu-img: root@testingbox:/tmp/test # qemu-img info input.qcow2 image: input.qcow2 file format: qcow2 virtual size: 50G (53687091200 bytes) disk size: 2.3G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true root@testingbox:/tmp/test # qemu-img info output.qcow2 image: output.qcow2 file format: qcow2 virtual size: 50G (53687091200 bytes) disk size: 789M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false Thanks!
Richard W.M. Jones
2016-Jun-17 08:11 UTC
Re: [libvirt-users] virt-sparsify changing the apparent-size of files
On Wed, Jun 15, 2016 at 10:20:54AM -0500, libvirt_users@skagitattic.com wrote:> Hello, > > I am curious why when using virt-sparsify the apparent-size of the file > gets reduced to the actual file size? Is there a way to get the > apparent-size left as the full size? > > In the man page for virt-sparsify it has a section on > "IMPORTANT NOTE ABOUT SPARSE OUTPUT IMAGES" and its output seems to > indicate what I would expect where "ls -lh" shows the full apparent size > and "du -sh" shows the actual size. But my actual results from testing > are different. > > In the following example input.qcow2 is before sparsify and output.qcow2 > is after. > > Doing ls reports full size for input but a small size for > the output: > root@testingbox:/tmp/test # ls -lh > total 3.1G > -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img > -rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2 > -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2What is file.img?> Same when asking du for apparent-size: > root@testingbox:/tmp/test # du -h --apparent-size * > 512M file.img > 51G input.qcow2 > 790M output.qcow2'du --apparent-size' and 'ls -l' are the same thing, so it's not really surprising that you see the same numbers here.> A normal du will show the actual size of the files: > root@testingbox:/tmp/test # du -h * > 0 file.img > 2.4G input.qcow2 > 790M output.qcow2 > > The only way I have found to check the full virtual size of the file > after virt-sparsify is with qemu-img: > root@testingbox:/tmp/test # qemu-img info input.qcow2 > image: input.qcow2 > file format: qcow2 > virtual size: 50G (53687091200 bytes) > disk size: 2.3G > cluster_size: 65536 > Format specific information: > compat: 1.1 > lazy refcounts: true > root@testingbox:/tmp/test # qemu-img info output.qcow2 > image: output.qcow2 > file format: qcow2 > virtual size: 50G (53687091200 bytes) > disk size: 789M > cluster_size: 65536 > Format specific information: > compat: 1.1 > lazy refcounts: falseIf I understand your question correctly, then everything is working as it should. For qcow2 files, the virtual size is not related to the "apparent" size, as you can easily prove: $ qemu-img create -f qcow2 huge.qcow2 1T Formatting 'huge.qcow2', fmt=qcow2 size=1099511627776 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 $ ls -lh huge.qcow2 -rw-r--r--. 1 rjones rjones 208K Jun 17 09:09 huge.qcow2 If you use raw files, then the file apparent size and virtual size are the same. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
libvirt_users@skagitattic.com
2016-Jun-22 05:40 UTC
Re: [libvirt-users] virt-sparsify changing the apparent-size of files
Hello Rich, Thanks for the reply and information.> What is file.img?Its simply a file I made while testing sparse files "truncate -s 512M file.img".> 'du --apparent-size' and 'ls -l' are the same thing, so it's not > really surprising that you see the same numbers here.Indeed, I expected those to be the same, just figured I would show them both for completeness.> For qcow2 files, the virtual size is not related to the "apparent" > size, as you can easily prove:So you are saying when I do the virt-sparsify its converting the image from raw to cow2? I studied the man page for virt-sparsify and tried again with the flag "--format raw" (output2.cow2). This output files looks as I first expected. # ls -lrh total 4.7G -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2 -rw-r--r-- 1 root root 51G Jun 21 18:34 output2.qcow2 -rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2 -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img # du -sh * 0 file.img 2.4G input.qcow2 1.6G output2.qcow2 790M output.qcow2 So I guess it was detecting the source wrong and doing a conversion? (As it says "If this is not specified, then the input format is used." in the man page) Also seems odd that it is 1.6G rather then the 790M of the output in qcow2 format. Does qcow2 do some compression or something? Is there some way I can see if a file is raw or cow2? When I did "qemu-img info" it reported both of them as cow2. I ran the initial install like the following and it ended up with the sparse file that reports full size with ls: "virt-install --name test-install --memory 512 -l 'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/' --disk size=50 --nographics -x "console=ttyS0"" I tried again and got the same result as the initial input file. So why is it naming it qcow2 if its actually raw? Or is there two way's of doing formatting inside the qcow2 file, one raw and the other called something else? Is there any reason to use raw vs non-raw? From some reading online it appears performance is better with raw files. Is there a downside to using raw? Thanks! Keelan On Fri, 17 Jun 2016 09:11:04 +0100 "Richard W.M. Jones" <rjones@redhat.com> wrote:> On Wed, Jun 15, 2016 at 10:20:54AM -0500, > libvirt_users@skagitattic.com wrote: > > Hello, > > > > I am curious why when using virt-sparsify the apparent-size of the > > file gets reduced to the actual file size? Is there a way to get > > the apparent-size left as the full size? > > > > In the man page for virt-sparsify it has a section on > > "IMPORTANT NOTE ABOUT SPARSE OUTPUT IMAGES" and its output seems to > > indicate what I would expect where "ls -lh" shows the full apparent > > size and "du -sh" shows the actual size. But my actual results > > from testing are different. > > > > In the following example input.qcow2 is before sparsify and > > output.qcow2 is after. > > > > Doing ls reports full size for input but a small size for > > the output: > > root@testingbox:/tmp/test # ls -lh > > total 3.1G > > -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img > > -rw-r--r-- 1 root root 51G Jun 14 22:30 input.qcow2 > > -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2 > > What is file.img? > > > Same when asking du for apparent-size: > > root@testingbox:/tmp/test # du -h --apparent-size * > > 512M file.img > > 51G input.qcow2 > > 790M output.qcow2 > > 'du --apparent-size' and 'ls -l' are the same thing, so it's not > really surprising that you see the same numbers here. > > > A normal du will show the actual size of the files: > > root@testingbox:/tmp/test # du -h * > > 0 file.img > > 2.4G input.qcow2 > > 790M output.qcow2 > > > > The only way I have found to check the full virtual size of the file > > after virt-sparsify is with qemu-img: > > root@testingbox:/tmp/test # qemu-img info input.qcow2 > > image: input.qcow2 > > file format: qcow2 > > virtual size: 50G (53687091200 bytes) > > disk size: 2.3G > > cluster_size: 65536 > > Format specific information: > > compat: 1.1 > > lazy refcounts: true > > root@testingbox:/tmp/test # qemu-img info output.qcow2 > > image: output.qcow2 > > file format: qcow2 > > virtual size: 50G (53687091200 bytes) > > disk size: 789M > > cluster_size: 65536 > > Format specific information: > > compat: 1.1 > > lazy refcounts: false > > If I understand your question correctly, then everything is working as > it should. > > For qcow2 files, the virtual size is not related to the "apparent" > size, as you can easily prove: > > $ qemu-img create -f qcow2 huge.qcow2 1T > Formatting 'huge.qcow2', fmt=qcow2 size=1099511627776 encryption=off > cluster_size=65536 lazy_refcounts=off refcount_bits=16 $ ls -lh > huge.qcow2 -rw-r--r--. 1 rjones rjones 208K Jun 17 09:09 huge.qcow2 > > If you use raw files, then the file apparent size and virtual size are > the same. > > Rich. >
Reasonably Related Threads
- Re: [libvirt-users] virt-sparsify changing the apparent-size of files
- virt-sparsify changing the apparent-size of files
- virt-sparsify changing the apparent-size of files
- Re: [libvirt-users] virt-sparsify changing the apparent-size of files
- Re: [libvirt-users] virt-sparsify changing the apparent-size of files