Libguestfs - Nov 2017 - Re: [Qemu-devel] [qemu-img] support for XVA

On 2017-11-15 21:24, Richard W.M. Jones wrote:
> On Wed, Nov 15, 2017 at 09:07:12PM +0100, Max Reitz wrote:
>> On 2017-11-15 21:06, Gandalf Corvotempesta wrote:
>>> 2017-11-15 20:59 GMT+01:00 Max Reitz <mreitz@redhat.com>:
>>>> Well, you can't just add support to qemu-img alone either. 
Every image
>>>> format supported by qemu-img is one supported by qemu as a
whole and
>>>> thus needs a proper block driver that needs to support random
accesses
>>>> as well.
>>>
>>> I was talking about qemu-img convert, just to convert an XVA image
to
>>> something different, in a single pass, without having to extract
the
>>> tar.
>>
>> I know, but that doesn't work.  qemu-img convert uses the normal
>> general-purpose block drivers for that.
> 
> In any case there's no need as qemu/qemu-img can already access files
> inside an uncompressed tarball using the offset/size support added to
> the raw driver exactly for this purpose:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03945.html
If that works, yes.  To me it doesn't look like XVA is just a single raw
image inside of a tarball, but I may very well be wrong, of course.

Max

On Wed, Nov 15, 2017 at 09:27:14PM +0100, Max Reitz
wrote:
> On 2017-11-15 21:24, Richard W.M. Jones wrote:
> > On Wed, Nov 15, 2017 at 09:07:12PM +0100, Max Reitz wrote:
> >> On 2017-11-15 21:06, Gandalf Corvotempesta wrote:
> >>> 2017-11-15 20:59 GMT+01:00 Max Reitz
<mreitz@redhat.com>:
> >>>> Well, you can't just add support to qemu-img alone
either.  Every image
> >>>> format supported by qemu-img is one supported by qemu as a
whole and
> >>>> thus needs a proper block driver that needs to support
random accesses
> >>>> as well.
> >>>
> >>> I was talking about qemu-img convert, just to convert an XVA
image to
> >>> something different, in a single pass, without having to
extract the
> >>> tar.
> >>
> >> I know, but that doesn't work.  qemu-img convert uses the
normal
> >> general-purpose block drivers for that.
> > 
> > In any case there's no need as qemu/qemu-img can already access
files
> > inside an uncompressed tarball using the offset/size support added to
> > the raw driver exactly for this purpose:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03945.html
> 
> If that works, yes.  To me it doesn't look like XVA is just a single
raw
> image inside of a tarball, but I may very well be wrong, of course.
Gandalf, is there an XVA file publically available (pref. not
too big) that we can look at?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

2017-11-15 21:29 GMT+01:00 Richard W.M. Jones
<rjones@redhat.com>:
> Gandalf, is there an XVA file publically available (pref. not
> too big) that we can look at?
I can try to provide one, but it's simple:

# tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
---------- 0/0           42353 1970-01-01 01:00 ova.xml
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000000.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000001.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000003.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000004.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000005.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000006.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000007.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000009.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000010.checksum


You can ignore the ova.xml and just use the "Ref:175" directory.
Inside the XVA you'll fine one "Ref" directory for each virtual
disk
(ref number is different for each disk)
Inside each Ref directory, you'll get tons of 1MB file, corrisponding
to the RAW image.
You have to merge these files in a single raw file with just an
exception: numbers are not sequential.
as you can see above, we have: 00000000, 00000001, 00000003

The 00000002 is missing, because it's totally blank. XenServer doesn't
export any empty block, thus it will skip the corrisponding 1MB file.
When building the raw image, you have to replace empty blocks with 1MB
full of zeros.

This is (in addition to the tar extract) the most time-consuming part.
Currently I'm rebuilding a 250GB image, with tons of files to be
merge.
If qemu-img can be patched to automatically convert this kind of
format, I can save about 3 hours (30 minutes for extracting the
tarball, and about 2 hours to merge 250-300GB image)