Libguestfs - Aug 2018 - Re: [PATCH] v2v: Add --print-estimate option to print source size estimate.

Am 15.08.2018 um 12:26 hat Richard W.M. Jones
geschrieben:
> On Tue, Aug 14, 2018 at 09:31:06PM +0300, Nir Soffer wrote:
> > On Tue, Aug 14, 2018 at 8:29 PM Richard W.M. Jones
<rjones@redhat.com>
> > wrote:
> > 
> > > This option prints the estimated size of the data that will be
copied
> > > from the source disk.
> > >
> > > For interest, the test prints:
> > >
> > > 3747840 ../test-data/phony-guests/windows.img
> > > Estimate: 3710976
> > >
> > 
> > Why not use qemu-img measure on the overlay?
> 
> Yes this would be better, but oddly it doesn't work properly when
> the output format is set to 'raw':
> 
>   /usr/bin/qemu-img 'measure' '-f' 'qcow2'
'-O' 'raw' '--output=human'
'/home/rjones/d/libguestfs/tmp/v2vovl62b7c2.qcow2'
>   required size: 6442450944
>   fully allocated size: 6442450944
> 
> However it's OK if the output format is set to 'qcow2':
> 
>   /usr/bin/qemu-img 'measure' '-f' 'qcow2'
'-O' 'qcow2' '--output=human'
'/home/rjones/d/libguestfs/tmp/v2vovla53d7c.qcow2'
>   required size: 1047986176
>   fully allocated size: 6443696128
> 
> I guess it ignores sparseness of raw images, but I can't see a way to
> specify that on the command line.
> 
> OTOH the qcow2 figure is probably a good enough guess for our purposes
> (ie. estimating how much data will be copied).
'qemu-img measure' calculates the resulting file size, not the number of
used blocks. I think this is because its main purpose is to create block
devices (like LVs) of the right size, so sparseness on a filesystem
level doesn't buy you anything.

Kevin

On Wed, Aug 15, 2018 at 1:55 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> (Adding Stefan who implemented the subcommand)
>
> On Wed, Aug 15, 2018 at 12:44:44PM +0200, Kevin Wolf wrote:
> > Am 15.08.2018 um 12:26 hat Richard W.M. Jones geschrieben:
> > > On Tue, Aug 14, 2018 at 09:31:06PM +0300, Nir Soffer wrote:
> > > > On Tue, Aug 14, 2018 at 8:29 PM Richard W.M. Jones <
> rjones@redhat.com>
> > > > wrote:
> > > >
> > > > > This option prints the estimated size of the data that
will be
> copied
> > > > > from the source disk.
> > > > >
> > > > > For interest, the test prints:
> > > > >
> > > > > 3747840 ../test-data/phony-guests/windows.img
> > > > > Estimate: 3710976
> > > > >
> > > >
> > > > Why not use qemu-img measure on the overlay?
> > >
> > > Yes this would be better, but oddly it doesn't work properly
when
> > > the output format is set to 'raw':
> > >
> > >   /usr/bin/qemu-img 'measure' '-f'
'qcow2' '-O' 'raw' '--output=human'
> '/home/rjones/d/libguestfs/tmp/v2vovl62b7c2.qcow2'
> > >   required size: 6442450944
> > >   fully allocated size: 6442450944
> > >
> > > However it's OK if the output format is set to
'qcow2':
> > >
> > >   /usr/bin/qemu-img 'measure' '-f'
'qcow2' '-O' 'qcow2'
> '--output=human'
'/home/rjones/d/libguestfs/tmp/v2vovla53d7c.qcow2'
> > >   required size: 1047986176
> > >   fully allocated size: 6443696128
> > >
> > > I guess it ignores sparseness of raw images, but I can't see
a way to
> > > specify that on the command line.
> > >
> > > OTOH the qcow2 figure is probably a good enough guess for our
purposes
> > > (ie. estimating how much data will be copied).
> >
> > 'qemu-img measure' calculates the resulting file size, not the
number of
> > used blocks. I think this is because its main purpose is to create
block
> > devices (like LVs) of the right size, so sparseness on a filesystem
> > level doesn't buy you anything.
>
> But if I run ‘qemu-img convert ... -O raw output.raw’ then output.raw
> will be a sparse file, and that's the file size I'd expect measure
to
> give us for "required size" (of course "fully allocated
size" would be
> the virtual size, and that is correct).
>
> It does look as if the block/raw-format.c:raw_measure function is wrong.
>
> In any case we can use the qcow2 estimate for our purposes as it will
> be near enough what we need (a rough estimate of the size of the copy).
>
Yes, using qcow2 as output format looks like a good estimate. In our tests
it is always less then 1.1 * virtual size.

Nir

On Wed, Aug 15, 2018 at 1:55 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> (Adding Stefan who implemented the subcommand)
>
> On Wed, Aug 15, 2018 at 12:44:44PM +0200, Kevin Wolf wrote:
> > Am 15.08.2018 um 12:26 hat Richard W.M. Jones geschrieben:
> > > On Tue, Aug 14, 2018 at 09:31:06PM +0300, Nir Soffer wrote:
> > > > On Tue, Aug 14, 2018 at 8:29 PM Richard W.M. Jones <
> rjones@redhat.com>
> > > > wrote:
> > > >
> > > > > This option prints the estimated size of the data that
will be
> copied
> > > > > from the source disk.
> > > > >
> > > > > For interest, the test prints:
> > > > >
> > > > > 3747840 ../test-data/phony-guests/windows.img
> > > > > Estimate: 3710976
> > > > >
> > > >
> > > > Why not use qemu-img measure on the overlay?
> > >
> > > Yes this would be better, but oddly it doesn't work properly
when
> > > the output format is set to 'raw':
> > >
> > >   /usr/bin/qemu-img 'measure' '-f'
'qcow2' '-O' 'raw' '--output=human'
> '/home/rjones/d/libguestfs/tmp/v2vovl62b7c2.qcow2'
> > >   required size: 6442450944
> > >   fully allocated size: 6442450944
> > >
> > > However it's OK if the output format is set to
'qcow2':
> > >
> > >   /usr/bin/qemu-img 'measure' '-f'
'qcow2' '-O' 'qcow2'
> '--output=human'
'/home/rjones/d/libguestfs/tmp/v2vovla53d7c.qcow2'
> > >   required size: 1047986176
> > >   fully allocated size: 6443696128
> > >
> > > I guess it ignores sparseness of raw images, but I can't see
a way to
> > > specify that on the command line.
> > >
> > > OTOH the qcow2 figure is probably a good enough guess for our
purposes
> > > (ie. estimating how much data will be copied).
> >
> > 'qemu-img measure' calculates the resulting file size, not the
number of
> > used blocks. I think this is because its main purpose is to create
block
> > devices (like LVs) of the right size, so sparseness on a filesystem
> > level doesn't buy you anything.
>
> But if I run ‘qemu-img convert ... -O raw output.raw’ then output.raw
> will be a sparse file, and that's the file size I'd expect measure
to
> give us for "required size" (of course "fully allocated
size" would be
> the virtual size, and that is correct).
>
> It does look as if the block/raw-format.c:raw_measure function is wrong.
>
Yes, it looks like the "required-size" value for raw output format is
not
very helpful. We could report there the number of bytes that would be
allocated after the image is copied.

>
> In any case we can use the qcow2 estimate for our purposes as it will
> be near enough what we need (a rough estimate of the size of the copy).
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
>

[ Cc: qemu-block ]

Am 15.08.2018 um 12:55 hat Richard W.M. Jones
geschrieben:
> (Adding Stefan who implemented the subcommand)
> 
> On Wed, Aug 15, 2018 at 12:44:44PM +0200, Kevin Wolf wrote:
> > Am 15.08.2018 um 12:26 hat Richard W.M. Jones geschrieben:
> > > On Tue, Aug 14, 2018 at 09:31:06PM +0300, Nir Soffer wrote:
> > > > On Tue, Aug 14, 2018 at 8:29 PM Richard W.M. Jones
<rjones@redhat.com>
> > > > wrote:
> > > > 
> > > > > This option prints the estimated size of the data that
will be copied
> > > > > from the source disk.
> > > > >
> > > > > For interest, the test prints:
> > > > >
> > > > > 3747840 ../test-data/phony-guests/windows.img
> > > > > Estimate: 3710976
> > > > >
> > > > 
> > > > Why not use qemu-img measure on the overlay?
> > > 
> > > Yes this would be better, but oddly it doesn't work properly
when
> > > the output format is set to 'raw':
> > > 
> > >   /usr/bin/qemu-img 'measure' '-f'
'qcow2' '-O' 'raw' '--output=human'
'/home/rjones/d/libguestfs/tmp/v2vovl62b7c2.qcow2'
> > >   required size: 6442450944
> > >   fully allocated size: 6442450944
> > > 
> > > However it's OK if the output format is set to
'qcow2':
> > > 
> > >   /usr/bin/qemu-img 'measure' '-f'
'qcow2' '-O' 'qcow2' '--output=human'
'/home/rjones/d/libguestfs/tmp/v2vovla53d7c.qcow2'
> > >   required size: 1047986176
> > >   fully allocated size: 6443696128
> > > 
> > > I guess it ignores sparseness of raw images, but I can't see
a way to
> > > specify that on the command line.
> > > 
> > > OTOH the qcow2 figure is probably a good enough guess for our
purposes
> > > (ie. estimating how much data will be copied).
> > 
> > 'qemu-img measure' calculates the resulting file size, not the
number of
> > used blocks. I think this is because its main purpose is to create
block
> > devices (like LVs) of the right size, so sparseness on a filesystem
> > level doesn't buy you anything.
> 
> But if I run ‘qemu-img convert ... -O raw output.raw’ then output.raw
> will be a sparse file, and that's the file size I'd expect measure
to
> give us for "required size" (of course "fully allocated
size" would be
> the virtual size, and that is correct).
> 
> It does look as if the block/raw-format.c:raw_measure function is wrong.
No, it is correct. Its output is what the file size will be. For raw
images, that is the same as the virtual disk size.

Not all blocks in the file will be actually allocated, but the file size
is what 'ls -l' prints, not what 'du' prints (for regular
files).

It becomes even clearer when you create LVs as the target. If you have a
10 GB image in which only the last 1 MB actually contains data, you
still need a 10 GB volume. You can't create a 1 MB volume and then store
data at an offset 10 GB - 1 MB, that would be way after the end of the
block device.
> In any case we can use the qcow2 estimate for our purposes as it will
> be near enough what we need (a rough estimate of the size of the copy).
I don't know what the exact purpose is, and it feels a bit hacky, but it
might be good enough.

I suppose what you really want is that 'qemu-img measure' provides
another number for the space taken by allocated blocks. (Probably
excluding metadata for non-raw formats? Might depend on the actual
purpose.)

Kevin