Libguestfs - Nov 2017 - Re: [Qemu-devel] [qemu-img] support for XVA

2017-11-15 23:55 GMT+01:00 Max Reitz
<mreitz@redhat.com>:
> https://xanclic.moe/convert-xva.rb -- does this work?
> (It seems to works on the two example images I found...)
>
> An example is in the code, you use it like this:
>
> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73

It doesn't work on huge images:

# time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2 disk1.qcow2
/usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
qemu-img (Errno::E2BIG)
from /usr/bin/convert-xva.rb:119:in `<main>'

real 9m41.414s
user 0m19.280s
sys 0m3.260s

On 2017-11-16 11:08, Gandalf Corvotempesta wrote:
> 2017-11-15 23:55 GMT+01:00 Max Reitz <mreitz@redhat.com>:
>> https://xanclic.moe/convert-xva.rb -- does this work?
>> (It seems to works on the two example images I found...)
>>
>> An example is in the code, you use it like this:
>>
>> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73
> 
> 
> It doesn't work on huge images:
> 
> # time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2 disk1.qcow2
> /usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
> qemu-img (Errno::E2BIG)
> from /usr/bin/convert-xva.rb:119:in `<main>'
> 
> real 9m41.414s
> user 0m19.280s
> sys 0m3.260s
Well, there is this:

https://github.com/XanClic/qemu.rb/blob/master/examples/convert-xva-online.rb

That should get around that restriction, but currently it is stupidly
slow (because it just issues all 1 MB copy operations simultaneously),
and it doesn't use qemu-img convert.  Instead, it uses a real qemu with
block jobs.  Ideally, you'd use it like this:

$ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva
Ref:73      81936 M
$ qemu-img create -f qcow2 [whatever options you need] disk1.qcow2 \
    81936M
Formatting 'disk1.qcow2', fmt=qcow2 size=85916123136 encryption=off
cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva \
    Ref:73 \
    '{"driver":"qcow2",
      "detect-zeroes":"unmap",
      "discard":"unmap",
     
"file":{"driver":"file","filename":"disk1.qcow2"}}'
Adding block devices...
Starting block jobs...
100.00 % of jobs completed

(Maybe I can get it faster.)

Max

I did something different, that will build a raw image directly from a
xenserver export, on the fly.
Compared the resulting file (via MD5) with xenmygrate.py and there is a match.

Currently, this is the faster way to convert a XenServer image to a
raw file. Don't need to wait
for export, tar extract and conversion. It does all of that, at the
same time during the VM export in a single pass.

If someone interested in testing it, it would be apreciated.

2017-11-16 15:02 GMT+01:00 Max Reitz
<mreitz@redhat.com>:
> On 2017-11-16 11:08, Gandalf Corvotempesta wrote:
>> 2017-11-15 23:55 GMT+01:00 Max Reitz <mreitz@redhat.com>:
>>> https://xanclic.moe/convert-xva.rb -- does this work?
>>> (It seems to works on the two example images I found...)
>>>
>>> An example is in the code, you use it like this:
>>>
>>> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva
Ref:73
>>
>>
>> It doesn't work on huge images:
>>
>> # time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2
disk1.qcow2
>> /usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
>> qemu-img (Errno::E2BIG)
>> from /usr/bin/convert-xva.rb:119:in `<main>'
>>
>> real 9m41.414s
>> user 0m19.280s
>> sys 0m3.260s
>
> Well, there is this:
>
>
https://github.com/XanClic/qemu.rb/blob/master/examples/convert-xva-online.rb
>
> That should get around that restriction, but currently it is stupidly
> slow (because it just issues all 1 MB copy operations simultaneously),
> and it doesn't use qemu-img convert.  Instead, it uses a real qemu with
> block jobs.  Ideally, you'd use it like this:
>
> $ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva
> Ref:73      81936 M
> $ qemu-img create -f qcow2 [whatever options you need] disk1.qcow2 \
>     81936M
> Formatting 'disk1.qcow2', fmt=qcow2 size=85916123136 encryption=off
> cluster_size=65536 lazy_refcounts=off refcount_bits=16
> $ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva \
>     Ref:73 \
>     '{"driver":"qcow2",
>       "detect-zeroes":"unmap",
>       "discard":"unmap",
>      
"file":{"driver":"file","filename":"disk1.qcow2"}}'
> Adding block devices...
> Starting block jobs...
> 100.00 % of jobs completed
>
> (Maybe I can get it faster.)
>
> Max
>