Libguestfs - Jun 2016 - Re: [libvirt-users] virt-sparsify changing the apparent-size of files

On Wed, Jun 15, 2016 at 10:20:54AM -0500, libvirt_users@skagitattic.com
wrote:
> Hello, 
> 
> I am curious why when using virt-sparsify the apparent-size of the file
> gets reduced to the actual file size?  Is there a way to get the
> apparent-size left as the full size?
> 
> In the man page for virt-sparsify it has a section on
> "IMPORTANT NOTE ABOUT SPARSE OUTPUT IMAGES" and its output seems
to
> indicate what I would expect where "ls -lh" shows the full
apparent size
> and "du -sh" shows the actual size.  But my actual results from
testing
> are different.
> 
> In the following example input.qcow2 is before sparsify and output.qcow2
> is after.
> 
> Doing ls reports full size for input but a small size for
> the output:
> root@testingbox:/tmp/test # ls -lh
> total 3.1G
> -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
> -rw-r--r-- 1 root root  51G Jun 14 22:30 input.qcow2
> -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
What is file.img?
> Same when asking du for apparent-size:
> root@testingbox:/tmp/test # du -h --apparent-size *
> 512M    file.img
> 51G     input.qcow2
> 790M    output.qcow2
'du --apparent-size' and 'ls -l' are the same thing, so it's
not
really surprising that you see the same numbers here.
> A normal du will show the actual size of the files:
> root@testingbox:/tmp/test # du -h *
> 0       file.img
> 2.4G    input.qcow2
> 790M    output.qcow2
> 
> The only way I have found to check the full virtual size of the file
> after virt-sparsify is with qemu-img:
> root@testingbox:/tmp/test # qemu-img info input.qcow2
> image: input.qcow2
> file format: qcow2
> virtual size: 50G (53687091200 bytes)
> disk size: 2.3G
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     lazy refcounts: true
> root@testingbox:/tmp/test # qemu-img info output.qcow2
> image: output.qcow2
> file format: qcow2
> virtual size: 50G (53687091200 bytes)
> disk size: 789M
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     lazy refcounts: false
If I understand your question correctly, then everything is working as
it should.

For qcow2 files, the virtual size is not related to the "apparent"
size, as you can easily prove:

$ qemu-img create -f qcow2 huge.qcow2 1T
Formatting 'huge.qcow2', fmt=qcow2 size=1099511627776 encryption=off
cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ ls -lh huge.qcow2
-rw-r--r--. 1 rjones rjones 208K Jun 17 09:09 huge.qcow2

If you use raw files, then the file apparent size and virtual size are
the same.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Wed, Jun 22, 2016 at 12:40:40AM -0500, libvirt_users@skagitattic.com
wrote:
> So you are saying when I do the virt-sparsify its converting the image
> from raw to [qcow2]?
No.  virt-sparsify will use the same input and output formats, unless
you use the --convert option.
> I studied the man page for virt-sparsify and tried again with
> the flag "--format raw" (output2.cow2).  This output files looks
as I
> first expected.
> 
> # ls -lrh
> total 4.7G
> -rw-r--r-- 1 root root 790M Jun 14 22:36 output.qcow2
> -rw-r--r-- 1 root root  51G Jun 21 18:34 output2.qcow2
> -rw-r--r-- 1 root root  51G Jun 14 22:30 input.qcow2
> -rw-r--r-- 1 root root 512M Jun 14 22:30 file.img
> # du -sh *
> 0       file.img
> 2.4G    input.qcow2
> 1.6G    output2.qcow2
> 790M    output.qcow2
>
> So I guess it was detecting the source wrong and doing a conversion?
> (As it says "If this is not specified, then the input format is
used."
> in the man page)
> 
> Also seems odd that it is 1.6G rather then the 790M of the output in
> qcow2 format.  Does qcow2 do some compression or something?
I'm really confused about what you actually are doing.

Delete what you've done and start from the beginning.  Describe
exactly how you created the guest.  Use 'qemu-img info' to show the
format of the input file.  Show precisely the virt-sparsify command
you are running.  And use 'qemu-img info' on the output file too.
> Is there any reason to use raw vs non-raw?  From some reading online it
> appears performance is better with raw files.  Is there a downside to
> using raw?
Raw has fewer features, especially no simple snapshotting or support
for backing files.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html