Libguestfs - Jun 2016 - Re: [libvirt-users] virt-sparsify changing the apparent-size of files

On Wed, Jun 22, 2016 at 05:56:09PM -0500, libvirt_users@skagitattic.com
wrote:
> > Delete what you've done and start from the beginning.  Describe
> > exactly how you created the guest.  Use 'qemu-img info' to
show the
> > format of the input file.  Show precisely the virt-sparsify command
> > you are running.  And use 'qemu-img info' on the output file
too.
> 
> Okay, here goes.  Start off with an install 
> 
>     root@testingbox:~ # virt-install --name testimage --memory 512 -l
>    
'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/'
>     --disk size=50 --nographics -x "console=ttyS0"             
> 
>     Starting install...
>     Retrieving file
>     MANIFEST...
>     | 3.3 kB     00:00 ... Retrieving file
>     linux...
>     | 6.0 MB     00:03 ... Retrieving file
>     initrd.gz...
>     |  29 MB     00:09 ... Allocating
>     'testimage.qcow2'
>     |  50 GB     00:00 Creating
>     domain...
>     |    0 B     00:00 Connected to domain testimage Escape character is
>     [    0.000000] Initializing cgroup subsys cpuset [    0.000000]
>     Initializing cgroup subsys cpu [    0.000000] Initializing cgroup
>     subsys cpuacct [    0.000000] Linux version 3.16.0-4-amd64
>     (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1)
>     ) #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
> 
> ....   Many more lines here and it turns interactive.  Suffice it to
> say I did a basic minimal install following the prompts.  Nothing
> special here.
> 
> This created an image that is 2.5G with a 50G apparent size and
> identifies as qcow2 as shown below.
> 
>     # ls -slh testimage.qcow2 
>     2.5G -rw-r--r-- 1 root root 51G Jun 22 20:22 testimage.qcow2
> 
>     # qemu-img info testimage.qcow2
>     image: testimage.qcow2
>     file format: qcow2
>     virtual size: 50G (53687091200 bytes)
>     disk size: 2.4G
>     cluster_size: 65536
>     Format specific information:
>         compat: 1.1
>         lazy refcounts: true
> 
> Now we sparsify (/bigtmp is on a larger partition then / to give
> virt-sparsify enough room to work with so it does not complain).
> 
>     root@testingbox: 08:59 PM # virt-sparsify testimage.qcow2
>     testimage1.qcow2 --tmp /bigtmp Input disk virtual size = 53687091200
>     bytes (50.0G) Create overlay file in /bigtmp to protect source
>     disk ... Examine source disk ...
>        100%
>
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
>     --:-- Fill free space in /dev/sda1 with zero ... 100%
>
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
>     00:00 Clearing Linux swap on /dev/sda5 ... 100%
>
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
>     --:-- Copy to destination and make sparse ...
> 
>     Sparsify operation completed with no errors.  Before deleting the
>     old disk, carefully check that the target disk boots and works
>     correctly. 
>     root@testingbox: 09:04 PM # 
> 
> This image reports still reports as qcow2 however the apparent and real
> size is the same.
> 
>     # ls -slh testimage1.qcow2 
>     1.1G -rw-r--r-- 1 root root 1.1G Jun 22 21:04 testimage1.qcow2
> 
>     # qemu-img info testimage1.qcow2
>     image: testimage1.qcow2
>     file format: qcow2
>     virtual size: 50G (53687091200 bytes)
>     disk size: 1.1G
>     cluster_size: 65536
>     Format specific information:
>         compat: 1.1
>         lazy refcounts: false
virt-sparsify saved about 1.3 GB of disk space (2.4G -> 1.1G).
> If we try it again but specify raw its MUCH faster
> 
>     root@testingbox: 09:26 PM # virt-sparsify testimage.qcow2
>     testimage2.qcow2 --tmp /bigtmp --format raw
This is incorrect usage.  You've told virt-sparsify that the input
image is raw.  But it's not raw, it's qcow2.

In any case, virt-sparsify opened the image as raw, couldn't make any
sense of it (it appears to virt-sparsify to be random data, not a disk
image), and so it cannot sparsify it properly.  This is "quicker" only
because virt-sparsify didn't do any sparsification of filesystems,
because it couldn't see the filesystems.
> This time it takes up more space and reports real and apparent size
> differently.  It still reports as qcow2 with qemu-img.
> 
>     # ls -slh testimage2.qcow2 
>     1.7G -rw-r--r-- 1 root root 51G Jun 22 21:27 testimage2.qcow2
>     # qemu-img info testimage2.qcow2
>     image: testimage2.qcow2
>     file format: qcow2
>     virtual size: 50G (53687091200 bytes)
>     disk size: 1.7G
Looks like virt-sparsify still found some clusters containing all
zeroes which it could trim, even though it didn't know how to read the
disk.

Anyway, the moral is, don't misinform virt-sparsify of the format
of your input image.

If what you really intended to do was to convert the format from qcow2
to raw, you should use '--convert raw' instead.
> The only things I see different from the finished files is that the raw
> one uses a bit more space, the apparent size is not set to the
> virtual size on the default sparsify one and the lazy refcounts is
> different.
> 
> Is there something more detailed then "qemu-img info" that I
could tell
> the difference with?
qemu-img info is the correct tool to use.
> > Raw has fewer features, especially no simple snapshotting or support
> > for backing files.
I wrote this thinking you were talking about actual raw format disks,
not about qcow2 but where you've misinformed virt-sparsify about the
format.
> Is simple snapshotting using qemu-img snapshot or something else?  I
> tried making a snapshot with both test images and they both appeared to
> work.
http://wiki.qemu.org/Documentation/CreateSnapshot
>     # qemu-img snapshot -c test  testimage1.qcow2 
>     # qemu-img snapshot -c test  testimage2.qcow2 
>     # qemu-img info  testimage2.qcow2 
>     image: testimage2.qcow2
>     file format: qcow2
>     virtual size: 50G (53687091200 bytes)
>     disk size: 1.7G
>     cluster_size: 65536
>     Snapshot list:
>     ID        TAG                 VM SIZE                DATE       VM
>      CLOCK 1         test                      0 2016-06-22 22:41:52
>      00:00:00.000 Format specific information:
>         compat: 1.1
>         lazy refcounts: true
This works because testimage2.qcow2 is still in qcow2 format.

Anyway, use the correct --format option.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Hello,

Thanks for all the information, I think my last remaining question is
why the image created with virt-install shows the full apparent size
and the image from virt-sparsify does not?  (The image testimage1.qcow2
created with default options).

As they are both qcow2 I would expect them both to have the same
behavior as far as apparent size goes.  Or is there an option I am
missing in virt-sparsify that would keep the full apparent size?  Or
perhaps one in virt-install to set apperent and real size the same?



On Thu, 23 Jun 2016 09:39:52 +0200
Andrea Bolognani <abologna@redhat.com> wrote:
> The '--format' option is to specify the image format for the
> input image. If you want the *output* image to be raw, you'll
> have to use '--convert raw'.
On Thu, 23 Jun 2016 08:52:35 +0100
"Richard W.M. Jones" <rjones@redhat.com>
wrote:
> This is incorrect usage.  You've told virt-sparsify that the input
> image is raw.  But it's not raw, it's qcow2.
That makes sense, I won't try and tell it the input is raw anymore
when its not.


Keelan



On Thu, 23 Jun 2016 08:52:35 +0100
"Richard W.M. Jones" <rjones@redhat.com> wrote:
> On Wed, Jun 22, 2016 at 05:56:09PM -0500,
> libvirt_users@skagitattic.com wrote:
> > > Delete what you've done and start from the beginning. 
Describe
> > > exactly how you created the guest.  Use 'qemu-img info'
to show
> > > the format of the input file.  Show precisely the virt-sparsify
> > > command you are running.  And use 'qemu-img info' on the
output
> > > file too.  
> > 
> > Okay, here goes.  Start off with an install 
> > 
> >     root@testingbox:~ # virt-install --name testimage --memory 512
> > -l
> >
'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/'
> > --disk size=50 --nographics -x "console=ttyS0"             
> > 
> >     Starting install...
> >     Retrieving file
> >     MANIFEST...
> >     | 3.3 kB     00:00 ... Retrieving file
> >     linux...
> >     | 6.0 MB     00:03 ... Retrieving file
> >     initrd.gz...
> >     |  29 MB     00:09 ... Allocating
> >     'testimage.qcow2'
> >     |  50 GB     00:00 Creating
> >     domain...
> >     |    0 B     00:00 Connected to domain testimage Escape
> > character is [    0.000000] Initializing cgroup subsys cpuset
> > [    0.000000] Initializing cgroup subsys cpu [    0.000000]
> > Initializing cgroup subsys cpuacct [    0.000000] Linux version
> > 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4
> > (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
> > 
> > ....   Many more lines here and it turns interactive.  Suffice it to
> > say I did a basic minimal install following the prompts.  Nothing
> > special here.
> > 
> > This created an image that is 2.5G with a 50G apparent size and
> > identifies as qcow2 as shown below.
> > 
> >     # ls -slh testimage.qcow2 
> >     2.5G -rw-r--r-- 1 root root 51G Jun 22 20:22 testimage.qcow2
> > 
> >     # qemu-img info testimage.qcow2
> >     image: testimage.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 2.4G
> >     cluster_size: 65536
> >     Format specific information:
> >         compat: 1.1
> >         lazy refcounts: true
> > 
> > Now we sparsify (/bigtmp is on a larger partition then / to give
> > virt-sparsify enough room to work with so it does not complain).
> > 
> >     root@testingbox: 08:59 PM # virt-sparsify testimage.qcow2
> >     testimage1.qcow2 --tmp /bigtmp Input disk virtual size > >
53687091200 bytes (50.0G) Create overlay file in /bigtmp to protect
> > source disk ... Examine source disk ...
> >        100%
> >
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> >     --:-- Fill free space in /dev/sda1 with zero ... 100%
> >
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> >     00:00 Clearing Linux swap on /dev/sda5 ... 100%
> >
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> >     --:-- Copy to destination and make sparse ...
> > 
> >     Sparsify operation completed with no errors.  Before deleting
> > the old disk, carefully check that the target disk boots and works
> >     correctly. 
> >     root@testingbox: 09:04 PM # 
> > 
> > This image reports still reports as qcow2 however the apparent and
> > real size is the same.
> > 
> >     # ls -slh testimage1.qcow2 
> >     1.1G -rw-r--r-- 1 root root 1.1G Jun 22 21:04 testimage1.qcow2
> > 
> >     # qemu-img info testimage1.qcow2
> >     image: testimage1.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 1.1G
> >     cluster_size: 65536
> >     Format specific information:
> >         compat: 1.1
> >         lazy refcounts: false  
> 
> virt-sparsify saved about 1.3 GB of disk space (2.4G -> 1.1G).
> 
> > If we try it again but specify raw its MUCH faster
> > 
> >     root@testingbox: 09:26 PM # virt-sparsify testimage.qcow2
> >     testimage2.qcow2 --tmp /bigtmp --format raw  
> 
> This is incorrect usage.  You've told virt-sparsify that the input
> image is raw.  But it's not raw, it's qcow2.
> 
> In any case, virt-sparsify opened the image as raw, couldn't make any
> sense of it (it appears to virt-sparsify to be random data, not a disk
> image), and so it cannot sparsify it properly.  This is "quicker"
only
> because virt-sparsify didn't do any sparsification of filesystems,
> because it couldn't see the filesystems.
> 
> > This time it takes up more space and reports real and apparent size
> > differently.  It still reports as qcow2 with qemu-img.
> > 
> >     # ls -slh testimage2.qcow2 
> >     1.7G -rw-r--r-- 1 root root 51G Jun 22 21:27 testimage2.qcow2
> >     # qemu-img info testimage2.qcow2
> >     image: testimage2.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 1.7G  
> 
> Looks like virt-sparsify still found some clusters containing all
> zeroes which it could trim, even though it didn't know how to read the
> disk.
> 
> Anyway, the moral is, don't misinform virt-sparsify of the format
> of your input image.
> 
> If what you really intended to do was to convert the format from qcow2
> to raw, you should use '--convert raw' instead.
> 
> > The only things I see different from the finished files is that the
> > raw one uses a bit more space, the apparent size is not set to the
> > virtual size on the default sparsify one and the lazy refcounts is
> > different.
> > 
> > Is there something more detailed then "qemu-img info" that I
could
> > tell the difference with?  
> 
> qemu-img info is the correct tool to use.
> 
> > > Raw has fewer features, especially no simple snapshotting or
> > > support for backing files.  
> 
> I wrote this thinking you were talking about actual raw format disks,
> not about qcow2 but where you've misinformed virt-sparsify about the
> format.
> 
> > Is simple snapshotting using qemu-img snapshot or something else?  I
> > tried making a snapshot with both test images and they both
> > appeared to work.  
> 
> http://wiki.qemu.org/Documentation/CreateSnapshot
> 
> >     # qemu-img snapshot -c test  testimage1.qcow2 
> >     # qemu-img snapshot -c test  testimage2.qcow2 
> >     # qemu-img info  testimage2.qcow2 
> >     image: testimage2.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 1.7G
> >     cluster_size: 65536
> >     Snapshot list:
> >     ID        TAG                 VM SIZE                DATE
> > VM CLOCK 1         test                      0 2016-06-22 22:41:52
> >      00:00:00.000 Format specific information:
> >         compat: 1.1
> >         lazy refcounts: true  
> 
> This works because testimage2.qcow2 is still in qcow2 format.
> 
> Anyway, use the correct --format option.
> 
> Rich.
>

On Thu, Jun 23, 2016 at 10:57:43AM -0500, libvirt_users@skagitattic.com
wrote:
> Thanks for all the information, I think my last remaining question is
> why the image created with virt-install shows the full apparent size
> and the image from virt-sparsify does not?  (The image testimage1.qcow2
> created with default options).
I don't really know what you mean by "full apparent size".  If you
are
talking about the output of "ls -l", then don't -- that number is
unrelated to the virtual size or the space consumed on disk.  Always
use "qemu-img info".

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW