Gandalf Corvotempesta
2017-Nov-15 20:41 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:> Gandalf, is there an XVA file publically available (pref. not > too big) that we can look at?I can try to provide one, but it's simple: # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50 ---------- 0/0 42353 1970-01-01 01:00 ova.xml ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000000 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000000.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000001 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000001.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000003 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000003.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000004 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000004.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000005 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000005.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000006 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000006.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000007 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000007.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000009 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000009.checksum ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000010 ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000010.checksum You can ignore the ova.xml and just use the "Ref:175" directory. Inside the XVA you'll fine one "Ref" directory for each virtual disk (ref number is different for each disk) Inside each Ref directory, you'll get tons of 1MB file, corrisponding to the RAW image. You have to merge these files in a single raw file with just an exception: numbers are not sequential. as you can see above, we have: 00000000, 00000001, 00000003 The 00000002 is missing, because it's totally blank. XenServer doesn't export any empty block, thus it will skip the corrisponding 1MB file. When building the raw image, you have to replace empty blocks with 1MB full of zeros. This is (in addition to the tar extract) the most time-consuming part. Currently I'm rebuilding a 250GB image, with tons of files to be merge. If qemu-img can be patched to automatically convert this kind of format, I can save about 3 hours (30 minutes for extracting the tarball, and about 2 hours to merge 250-300GB image)
Richard W.M. Jones
2017-Nov-15 21:05 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
On Wed, Nov 15, 2017 at 09:41:20PM +0100, Gandalf Corvotempesta wrote:> 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>: > > Gandalf, is there an XVA file publically available (pref. not > > too big) that we can look at? > > I can try to provide one, but it's simple: > > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50 > ---------- 0/0 42353 1970-01-01 01:00 ova.xml > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000000 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000000.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000001 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000001.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000003 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000003.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000004 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000004.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000005 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000005.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000006 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000006.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000007 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000007.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000009 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000009.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000010 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000010.checksum > > > You can ignore the ova.xml and just use the "Ref:175" directory. > Inside the XVA you'll fine one "Ref" directory for each virtual disk > (ref number is different for each disk) > Inside each Ref directory, you'll get tons of 1MB file, corrisponding > to the RAW image. > You have to merge these files in a single raw file with just an > exception: numbers are not sequential. > as you can see above, we have: 00000000, 00000001, 00000003 > > The 00000002 is missing, because it's totally blank. XenServer doesn't > export any empty block, thus it will skip the corrisponding 1MB file. > When building the raw image, you have to replace empty blocks with 1MB > full of zeros. > > This is (in addition to the tar extract) the most time-consuming part. > Currently I'm rebuilding a 250GB image, with tons of files to be > merge. > If qemu-img can be patched to automatically convert this kind of > format, I can save about 3 hours (30 minutes for extracting the > tarball, and about 2 hours to merge 250-300GB image)I guess the nbdkit approach would be better given the multiple and missing files within the tar file. You'll have to use ‘tar tRvf file.xva’ to extract the offset and size of each file. (See the function ‘find_file_in_tar’ in virt-v2v source for exactly how). This will give you one offset/size/filename tuple for each file. The plugin will then simply need to calculate which file to access to resolve each virtual file range (or substitute zeroes for missing files). Note that nbdkit lets you write plugins in high-level languages like Perl or Python which would be ideally suited to this kind of task. You can then use "captive nbdkit" (see the manual) like this: nbdkit -U - \ perl script=/path/to/xva-parser.pl file=/path/to/file.xva \ --run 'qemu-img convert $nbd -O qcow2 output.qcow2' At no point does the tar file need to be fully unpacked and it should all be reasonably efficient. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Richard W.M. Jones
2017-Nov-15 21:15 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
On Wed, Nov 15, 2017 at 09:05:00PM +0000, Richard W.M. Jones wrote:> This will give you one offset/size/filename tuple for each file. The > plugin will then simply need to calculate which file to access to > resolve each virtual file range (or substitute zeroes for missing > files).By which I mean, of course, that it accesses the offset/size directly by opening the tar file. The file doesn't have to be extracted. I'm still interested if you have a small XVA example you can share. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On 2017-11-15 21:41, Gandalf Corvotempesta wrote:> 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>: >> Gandalf, is there an XVA file publically available (pref. not >> too big) that we can look at? > > I can try to provide one, but it's simple: > > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50 > ---------- 0/0 42353 1970-01-01 01:00 ova.xml > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000000 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000000.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000001 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000001.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000003 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000003.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000004 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000004.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000005 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000005.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000006 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000006.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000007 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000007.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000009 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000009.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000010 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000010.checksum > > > You can ignore the ova.xml and just use the "Ref:175" directory. > Inside the XVA you'll fine one "Ref" directory for each virtual disk > (ref number is different for each disk) > Inside each Ref directory, you'll get tons of 1MB file, corrisponding > to the RAW image. > You have to merge these files in a single raw file with just an > exception: numbers are not sequential. > as you can see above, we have: 00000000, 00000001, 00000003 > > The 00000002 is missing, because it's totally blank. XenServer doesn't > export any empty block, thus it will skip the corrisponding 1MB file. > When building the raw image, you have to replace empty blocks with 1MB > full of zeros. > > This is (in addition to the tar extract) the most time-consuming part. > Currently I'm rebuilding a 250GB image, with tons of files to be > merge. > If qemu-img can be patched to automatically convert this kind of > format, I can save about 3 hours (30 minutes for extracting the > tarball, and about 2 hours to merge 250-300GB image)Hmmm... OK, so as Rich said, you can use the raw driver with an offset and a size. And you can use the null-co driver to emulate holes. And qemu-img convert supports concatenation. Taking all of that together, it should be possible to create an input "file" per 1 MB block and concatenate all of those with qemu-img convert. The only issue is that it'll get you a *really* long command line. (And you'll still have to gunzip the archive before, of course.) I'll try to write a script, but of course it's hart without an example... Max
Gandalf Corvotempesta
2017-Nov-15 21:42 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
I'm thinking on how to prove you a sample XVA I have to create (and populate) a VM because an empty image will result in an empty XVA And a VM is 300-400Mb as minimum Il 15 nov 2017 10:30 PM, "Max Reitz" <mreitz@redhat.com> ha scritto:> On 2017-11-15 21:41, Gandalf Corvotempesta wrote: > > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>: > >> Gandalf, is there an XVA file publically available (pref. not > >> too big) that we can look at? > > > > I can try to provide one, but it's simple: > > > > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50 > > ---------- 0/0 42353 1970-01-01 01:00 ova.xml > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000000 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000000.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000001 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000001.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000003 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000003.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000004 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000004.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000005 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000005.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000006 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000006.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000007 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000007.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000009 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000009.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000010 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000010.checksum > > > > > > You can ignore the ova.xml and just use the "Ref:175" directory. > > Inside the XVA you'll fine one "Ref" directory for each virtual disk > > (ref number is different for each disk) > > Inside each Ref directory, you'll get tons of 1MB file, corrisponding > > to the RAW image. > > You have to merge these files in a single raw file with just an > > exception: numbers are not sequential. > > as you can see above, we have: 00000000, 00000001, 00000003 > > > > The 00000002 is missing, because it's totally blank. XenServer doesn't > > export any empty block, thus it will skip the corrisponding 1MB file. > > When building the raw image, you have to replace empty blocks with 1MB > > full of zeros. > > > > This is (in addition to the tar extract) the most time-consuming part. > > Currently I'm rebuilding a 250GB image, with tons of files to be > > merge. > > If qemu-img can be patched to automatically convert this kind of > > format, I can save about 3 hours (30 minutes for extracting the > > tarball, and about 2 hours to merge 250-300GB image) > > Hmmm... OK, so as Rich said, you can use the raw driver with an offset > and a size. And you can use the null-co driver to emulate holes. And > qemu-img convert supports concatenation. > > Taking all of that together, it should be possible to create an input > "file" per 1 MB block and concatenate all of those with qemu-img > convert. The only issue is that it'll get you a *really* long command > line. > > (And you'll still have to gunzip the archive before, of course.) > > I'll try to write a script, but of course it's hart without an example... > > Max > >
Richard W.M. Jones
2017-Nov-15 21:51 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
On Wed, Nov 15, 2017 at 10:30:31PM +0100, Max Reitz wrote:> Hmmm... OK, so as Rich said, you can use the raw driver with an offset > and a size. And you can use the null-co driver to emulate holes. And > qemu-img convert supports concatenation.Nice, I didn't know qemu-img convert could concatenate :-) So that's two distinct ways to do this, neither needing any qemu modifications. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Richard W.M. Jones
2017-Nov-16 10:01 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
Here's my solution, as a nbdkit plugin written in Perl.
As with Max's solution I don't bother to parse the virtual size out of
the XML file, so you need to specify that on the command line
otherwise the disk will be truncated to the largest extent stored in
the file. Also the ‘.xva’ file must not be compressed.
$ nbdkit perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296
$ guestfish --ro --format=raw -a nbd://localhost -i
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.
Type: 'help' for help on commands
'man' to read the manual
'quit' to quit the shell
Operating system: 8.2
/dev/sda1 mounted on /
><fs> ll /
total 100
drwxr-xr-x 22 root root 4096 Jan 8 2016 .
drwxr-xr-x 19 root root 4096 Nov 16 09:50 ..
drwxrwxr-x 2 root root 4096 Jan 8 2016 bin
drwxr-xr-x 3 root root 4096 Jan 8 2016 boot
drwxr-xr-x 4 root root 4096 Jan 8 2016 dev
drwxr-xr-x 87 root root 4096 Jan 8 2016 etc
drwxr-xr-x 3 root root 4096 Jan 8 2016 home
lrwxrwxrwx 1 root root 31 Jan 8 2016 initrd.img ->
/boot/initrd.img-3.16.0-4-amd64
drwxr-xr-x 14 root root 4096 Jan 8 2016 lib
drwxr-xr-x 2 root root 4096 Jan 8 2016 lib64
drwx------ 2 root root 16384 Jan 8 2016 lost+found
drwxr-xr-x 3 root root 4096 Jan 8 2016 media
drwxr-xr-x 2 root root 4096 Jan 8 2016 mnt
drwxr-xr-x 2 root root 4096 Jan 8 2016 opt
drwxr-xr-x 2 root root 4096 May 4 2015 proc
drwx------ 2 root root 4096 Jan 8 2016 root
drwxr-xr-x 2 root root 4096 Jan 8 2016 run
drwxr-xr-x 2 root root 4096 Jan 8 2016 sbin
drwxr-xr-x 2 root root 4096 Jan 8 2016 srv
drwxr-xr-x 2 root root 4096 Apr 6 2015 sys
drwxrwxrwt 7 root root 4096 Jan 8 2016 tmp
drwxr-xr-x 10 root root 4096 Jan 8 2016 usr
drwxr-xr-x 11 root root 4096 Jan 8 2016 var
lrwxrwxrwx 1 root root 27 Jan 8 2016 vmlinuz ->
boot/vmlinuz-3.16.0-4-amd64
I even managed to boot the Debian 8 guest from the sample .xva file:
$ qemu-system-x86_64 -cpu host -machine accel=kvm:tcg -m 2048 -drive
file=nbd:localhost:10809,format=raw,if=virtio,snapshot=on
although it was pretty slow ...
As mentioned before you can use this to do a qemu-img convert using
captive nbdkit:
$ nbdkit -U - \
perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \
--run 'qemu-img convert -f raw $nbd -O qcow2 /var/tmp/output.qcow2
-p'
HTH,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
--7AUc2qLy4jB3hD7Z
Content-Type: application/x-perl
Content-Disposition: attachment; filename="xva-reader.pl"
Content-Transfer-Encoding: quoted-printable
use strict;=0A=0Amy %config =3D ();=0Amy %block =3D ();=0A=0A# XXX Should parse
the virtual size from size from the XML document.=0A# Instead we estimate the
size here, and the user can override=0A# it by setting the optional size=3D...
parameter.=0Amy $estimated_size;=0A=0A# Does XVA define this? We might need to
read it from the file=0A# somehow.=0Amy $block_size =3D 1048576;=0A=0Asub
config=0A{=0A my $key =3D shift;=0A my $value =3D shift;=0A
$config{$key} =3D $value;=0A}=0A=0Asub config_complete=0A{=0A die "no
file=3D... parameter given" unless exists $config{file};=0A=0A my $file
=3D $config{file};=0A $estimated_size =3D 0;=0A=0A # Parse the XVA file
for offsets.=0A # XXX Does not work if the XVA contains multiple disk images.
We=0A # should parse the 'Ref:NN' field too.=0A open FILE,
"tar tRvf $file |" or die "$file: $!";=0A while
(<FILE>) {=0A if (m{^block (\d+): \S+
\d+/\d+\s+(\d+)\s.*/0*(\d+)$}) {=0A my $offset =3D $1; # block
offset in tar file=0A my $size =3D $2; # size in bytes in tar
file=0A my $filename =3D $3; # filename in tar file, without leading
0s=0A my $byte_offset =3D (1 + $offset) * 512; # byte offset in tar
file=0A $block{$filename} =3D { size =3D> $size, offset =3D>
$byte_offset };=0A #printf ("%s %d %d\n", $filename,
$byte_offset, $size);=0A=0A if ($byte_offset + $size >
$estimated_size) {=0A $estimated_size =3D $byte_offset +
$size;=0A }=0A }=0A }=0A close FILE;=0A}=0A=0Asub
open=0A{=0A my $readonly =3D shift;=0A my $file =3D $config{file};=0A
open (my $fh, "<", $file) or die "$file: $!";=0A my $h
=3D { fh =3D> $fh };=0A return $h;=0A}=0A=0Asub close=0A{=0A my $h =3D
shift;=0A close ($h->{fh});=0A}=0A=0Asub get_size=0A{=0A my $h =3D
shift;=0A my $i =3D $estimated_size;=0A # Allow user to override the
estimate by setting size=3D parameter.=0A $i =3D $config{size} if exists
$config{size};=0A return $i;=0A}=0A=0Asub read_block=0A{=0A my $h =3D
shift;=0A my $bnum =3D shift;=0A my $offset =3D $block{$bnum}{offset};=0A
my $size =3D $block{$bnum}{size};=0A seek ($h->{fh}, $offset, 0);=0A my
$ret =3D "";=0A my $i =3D 0;=0A while ($size > 0) {=0A
my $n =3D read ($h->{fh}, $ret, $size, $i);=0A if ($n =3D=3D 0) {=0A
die "unexpected end of file: block number $bnum (offset $offset, size
$size)"=0A }=0A $i +=3D $n;=0A $size -=3D $n;=0A
}=0A return $ret;=0A}=0A=0Asub pread=0A{=0A my $h =3D shift;=0A my
$count =3D shift;=0A my $offset =3D shift;=0A=0A my $buf =3D
"";=0A my $bufi =3D 0;=0A=0A while ($count > 0) {=0A #
Which block, if any, contains $offset?=0A my $bnum =3D int ($offset /
$block_size);=0A my $boffs =3D $offset % $block_size;=0A #print
"offset =3D $offset, bnum =3D $bnum, boffs =3D $boffs\n";=0A=0A
# n =3D number of bytes that can be read from this block.=0A my $n =3D
$block_size - $boffs;=0A $n =3D $count if $n > $count;=0A=0A if
(exists $block{$bnum}) {=0A my $bdata =3D read_block ($h, $bnum);=0A
$buf .=3D substr $bdata, $boffs, $n;=0A }=0A else {=0A
# Virtual zero block.=0A $buf .=3D "\0" x $n;=0A
}=0A=0A $count -=3D $n;=0A }=0A=0A return $buf;=0A}=0A
--7AUc2qLy4jB3hD7Z--
Gandalf Corvotempesta
2017-Nov-16 10:07 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
2017-11-16 11:01 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:> As mentioned before you can use this to do a qemu-img convert using > captive nbdkit: > > $ nbdkit -U - \ > perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \ > --run 'qemu-img convert -f raw $nbd -O qcow2 /var/tmp/output.qcow2 -p'What if XVA is hosting 2 or more partitions ? Which partition are you converting with this command ?
Tomáš Golembiovský
2017-Nov-16 12:56 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
On Wed, 15 Nov 2017 21:41:20 +0100 Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:> 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>: > > Gandalf, is there an XVA file publically available (pref. not > > too big) that we can look at? > > I can try to provide one, but it's simple: > > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50 > ---------- 0/0 42353 1970-01-01 01:00 ova.xml > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000000 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000000.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000001 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000001.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000003 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000003.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000004 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000004.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000005 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000005.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000006 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000006.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000007 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000007.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000009 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000009.checksum > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000010 > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000010.checksum > > > You can ignore the ova.xml and just use the "Ref:175" directory. > Inside the XVA you'll fine one "Ref" directory for each virtual disk > (ref number is different for each disk) > Inside each Ref directory, you'll get tons of 1MB file, corrisponding > to the RAW image. > You have to merge these files in a single raw file with just an > exception: numbers are not sequential. > as you can see above, we have: 00000000, 00000001, 00000003 > > The 00000002 is missing, because it's totally blank. XenServer doesn't > export any empty block, thus it will skip the corrisponding 1MB file. > When building the raw image, you have to replace empty blocks with 1MB > full of zeros. > > This is (in addition to the tar extract) the most time-consuming part. > Currently I'm rebuilding a 250GB image, with tons of files to be > merge. > If qemu-img can be patched to automatically convert this kind of > format, I can save about 3 hours (30 minutes for extracting the > tarball, and about 2 hours to merge 250-300GB image) >Hi, I like what Richard and Max came up with. Pretty nifty solutions. While there's nothing wrong with them I decided to take my own shot at it. Since the blocks in tar file are pieces of raw image there is no conversion happening. All in all it's just moving bytes from one place to another. That means there shouldn't be a need for any heavy machinery, right? :) Attached is a shell script that uses dd to do the byte-shuffling. I'm pretty sure this could even be safely parallelized by running multiple instances of dd at the same time (e.g. with xargs). But I did not try that. Tomas -- Tomáš Golembiovský <tgolembi@redhat.com>
Tomáš Golembiovský
2017-Nov-16 13:07 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
On Thu, 16 Nov 2017 13:56:16 +0100 Tomáš Golembiovský <tgolembi@redhat.com> wrote:> On Wed, 15 Nov 2017 21:41:20 +0100 > Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote: > > > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>: > > > Gandalf, is there an XVA file publically available (pref. not > > > too big) that we can look at? > > > > I can try to provide one, but it's simple: > > > > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50 > > ---------- 0/0 42353 1970-01-01 01:00 ova.xml > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000000 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000000.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000001 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000001.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000003 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000003.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000004 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000004.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000005 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000005.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000006 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000006.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000007 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000007.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000009 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000009.checksum > > ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000010 > > ---------- 0/0 40 1970-01-01 01:00 Ref:175/00000010.checksum > > > > > > You can ignore the ova.xml and just use the "Ref:175" directory. > > Inside the XVA you'll fine one "Ref" directory for each virtual disk > > (ref number is different for each disk) > > Inside each Ref directory, you'll get tons of 1MB file, corrisponding > > to the RAW image. > > You have to merge these files in a single raw file with just an > > exception: numbers are not sequential. > > as you can see above, we have: 00000000, 00000001, 00000003 > > > > The 00000002 is missing, because it's totally blank. XenServer doesn't > > export any empty block, thus it will skip the corrisponding 1MB file. > > When building the raw image, you have to replace empty blocks with 1MB > > full of zeros. > > > > This is (in addition to the tar extract) the most time-consuming part. > > Currently I'm rebuilding a 250GB image, with tons of files to be > > merge. > > If qemu-img can be patched to automatically convert this kind of > > format, I can save about 3 hours (30 minutes for extracting the > > tarball, and about 2 hours to merge 250-300GB image) > > > > Hi, > > I like what Richard and Max came up with. Pretty nifty solutions. > While there's nothing wrong with them I decided to take my own shot at > it. Since the blocks in tar file are pieces of raw image there is no > conversion happening. All in all it's just moving bytes from one place > to another. That means there shouldn't be a need for any heavy > machinery, right? :) > > Attached is a shell script that uses dd to do the byte-shuffling. > I'm pretty sure this could even be safely parallelized by running > multiple instances of dd at the same time (e.g. with xargs). But I did > not try that.Oh yes, and one more thing: as with the other solutions I didn't bother reading the XML for the target size. So resize may be necessary afterwards. Tomas -- Tomáš Golembiovský <tgolembi@redhat.com>
Richard W.M. Jones
2017-Nov-16 13:08 UTC
Re: [Libguestfs] [Qemu-devel] [qemu-img] support for XVA
I think it's even possible to modify that shell script to pipe into nbdkit and from there to ‘qemu-img convert’. I'm too lazy to actually do that right now, but the basic idea is here: https://rwmj.wordpress.com/2014/10/14/streaming-nbd-server/ You'll probably have to add ‘-m 1’ to the qemu-img convert command line. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org