Libguestfs - Mar 2022 - Real size vs virtual size image question?

Hello All,

I have been doing some reading and it seems that for some images created
that the "real" size is not the same as the reported
"virtual" size to the
filesystem.  There actually is a name for this, but I cannot remember what
it was now and think that I ran across it in my reading on the Libguestfs
site somewhere.

Anyway,  I have some images that have actual sizes around 4 MB but are
reporting sizes of 1 TB.

I would like to compress and zip up these images but based upon the 4 MB
size and not the 1 TB size.

Is there any simple tool that will do this, or perhaps some source code
that I can compile into a tool or library that will do this as I ultimately
need some simple C/C++ or Golang solution to compress and decompress these
files properly?

Any ideas or suggestions would be greatly appreciated.

Best Regards,
Lonnie
Email: Lonnie.Cumberland at Outstep.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20220314/64c1df5a/attachment.htm>

On Mon, Mar 14, 2022 at 01:14:27PM -0400, Lonnie Cumberland
wrote:
> Hello All,
> 
> I have been doing some reading and it seems that for some images created
that
> the "real" size is not the same as the reported
"virtual" size to the
> filesystem.? There actually is a name for this, but I cannot remember what
it
> was now and think that I ran across it in my reading on the Libguestfs site
> somewhere.
> 
> Anyway,? I have some images that have actual sizes around 4 MB but are
> reporting sizes of 1 TB.
Sure ...

  $ qemu-img create -f qcow2 test.qcow2 1T
  Formatting 'test.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off
compression_type=zlib size=1099511627776 lazy_refcounts=off refcount_bits=16

Real size about 200K:

  $ ls -lh test.qcow2 
  -rw-r--r--. 1 rjones rjones 208K Mar 14 17:38 test.qcow2

Virtual size 1TB:

  $ qemu-img info test.qcow2 
  image: test.qcow2
  file format: qcow2
  virtual size: 1 TiB (1099511627776 bytes)   <---
  disk size: 1 MiB

There's no magic here, the qcow2 format is simply able to ignore all
the blank sectors in the image, and even if the image contained some
data it would map the sectors efficiently into a smaller file.

 
https://web.archive.org/web/20100221075914/https://people.gnome.org/~markmc/qcow-image-format.html

libguestfs has several APIs to get the virtual size.
guestfs_disk_virtual_size is just a wrapper around qemu-img info:

  $ guestfish --ro disk-virtual-size test.qcow2
  1099511627776

guestfs_blockdev_getsize64 works from inside the appliance and can be
used to get the size of whole disks, partitions, etc within the image:

  $ guestfish --ro -a test.qcow2 run : blockdev-getsize64 /dev/sda
  1099511627776

We don't have an API to get the real size because you can just read it
using normal libc APIs like stat(2).
> I would like to compress and zip up these images but based upon the
> 4 MB size and not the 1 TB size.
>
> Is there any simple tool that will do this, or perhaps some source
> code that I can compile into a tool or library that will do this as
> I ultimately need some simple C/C++ or Golang solution to?compress
> and decompress these files properly?
I think we need a bit of clarity about what exactly you're trying to
achieve here.  What does "compress ... these images but based upon the
[real] size and not the [virtual] size" mean?

nbdkit xz-filter + file plugin can do _something_, whether it's what
you need or not I'm not sure:

https://libguestfs.org/nbdkit-xz-filter.1.html

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org