search for: trvf

...r, I'm a bit worried about how it would work if the tarfile itself includes a file named 'exportname' in the top directory of the tarfile. A quick test confirms my worry: $ cd /tmp $ touch exportname $ tar cf f.tar exportname $ tar tf f.tar exportname $ LANG=C tar --no-auto-compress -tRvf f.tar exportname block 0: -rw-rw-r-- eblake/eblake 0 2020-07-06 13:37 exportname block 1: ** Block of NULs ** $ LANG=C tar --no-auto-compress -tRvf f.tar ./exportname block 1: ** Block of NULs ** tar: ./exportname: Not found in archive tar: Exiting with failure status due to previous errors so...

...o find some way to read the underlying plugin to feed a pipeline to tar to decode what the file contains (we can't just hand the file to tar, but instead have to feed it data through stdin; but the amount of work is no different: tar really does have to read the entire image during 'tar tRvf'). - There would no longer be a tar=... parameter, rather that is the role of the plugin - We'd have to rename the file=... parameter to something that won't conflict with the most common use of having the file plugin serve the tar file - But once we've done the .get_ready scan o...

...Construct the tar command to examine the tar file. */ > >+ fp = open_memstream (&cmd, &len); > >+ if (fp == NULL) { > >+ nbdkit_error ("open_memstream: %m"); > >+ return -1; > >+ } > >+ fprintf (fp, "LANG=C tar --no-auto-compress -tRvf "); > > Use of --no-auto-compress is specific to GNU tar, do we care? I copied that from the existing Perl plugin, but note that -R is also not supported by non-GNU tar. > Should we allow a 'tar=/path/to/tar' parameter during .config to > allow a user to point to an alte...

Originally an attempt to fix: https://bugzilla.redhat.com/show_bug.cgi?id=1570407 However this isn't a complete fix. The OVA supplied doesn't even conform to VMware's own "specification" (I use the word loosely). The OVF inside the OVA references the disk.vmdk file, but the OVA doesn't contain that disk.vmdk file, only a snapshot called disk.vmdk.000000000. Therefore

...256]; > + > + /* Construct the tar command to examine the tar file. */ > + fp = open_memstream (&cmd, &len); > + if (fp == NULL) { > + nbdkit_error ("open_memstream: %m"); > + return -1; > + } > + fprintf (fp, "LANG=C tar --no-auto-compress -tRvf "); Using -R is nice, but is block size documented? Also --block-number would be nicer. > + shell_quote (tarfile, fp); This is questionable. Why use string and quote the string instead of using argv directly? > + fputc (' ', fp); > + shell_quote (file, fp); > + if...

For context see these threads: https://lists.gnu.org/archive/html/qemu-discuss/2020-06/threads.html#00053 https://lists.gnu.org/archive/html/qemu-block/2020-06/threads.html#01496 Rich.

...doc ~fmt:JSON.Compact json)) + (if verbose () then "" else " >/dev/null 2>&1") in + + let r = Sys.command cmd in + + Unix.unlink tmp1; + Unix.unlink tmp2; + + r = 0 let find_file_in_tar tar filename = let lines = external_command (sprintf "tar tRvf %s" (Filename.quote tar)) in diff --git a/v2v/utils.mli b/v2v/utils.mli index eb47d03..b75baa7 100644 --- a/v2v/utils.mli +++ b/v2v/utils.mli @@ -51,10 +51,9 @@ val du : string -> int64 This can raise either [Failure] or [Invalid_argument] in case of errors. *) -val qemu_img_ver...

On Sun, Jun 28, 2020 at 04:46:59PM +0300, Nir Soffer wrote: > On Sun, Jun 28, 2020 at 4:03 PM Richard W.M. Jones <rjones@redhat.com> wrote: > ... > > + fprintf (fp, "LANG=C tar --no-auto-compress -tRvf "); > > Using -R is nice, but is block size documented? > > Also --block-number would be nicer. Yes the block size is always 512 bytes: https://www.gnu.org/software/tar/manual/html_node/Blocking.html By the way, FreeBSD tar will not work (and presumably other non-GNU tar), s...

...ed to automatically convert this kind of > format, I can save about 3 hours (30 minutes for extracting the > tarball, and about 2 hours to merge 250-300GB image) I guess the nbdkit approach would be better given the multiple and missing files within the tar file. You'll have to use ‘tar tRvf file.xva’ to extract the offset and size of each file. (See the function ‘find_file_in_tar’ in virt-v2v source for exactly how). This will give you one offset/size/filename tuple for each file. The plugin will then simply need to calculate which file to access to resolve each virtual file range...

v1 -> v2: - Use 'qemu-img info' which avoids one of the temporary files (thanks Tomáš). - Add a unit test. Unfortunately when I run 'make -C v2v check' I don't think this code is being run at all, so there may be something else going on here which I don't understand. Rich.

Not tested yet, am running the tests now. Rich.

...mpress with gzip or xz), zip") ova ) @@ -135,6 +146,49 @@ object loop [dir] in + (* Find file in [tar] archive and return at which byte it starts and how + * long it is. + *) + let find_file_in_tar tar filename = + let cmd1 = [ "tar"; "tRvf"; tar ] in + let cmd2 = [ "awk"; sprintf + "$8 == \"%s\" {print substr($2, 1, index($2, \":\")-1), $5}" + filename ] + in + let lines = + external_command ((stringify_args cmd1) ^ " | " ^ (stringify_args cmd2...

...; unless exists $config{file};=0A=0A m= y $file =3D $config{file};=0A $estimated_size =3D 0;=0A=0A # Parse th= e XVA file for offsets.=0A # XXX Does not work if the XVA contains multi= ple disk images. We=0A # should parse the 'Ref:NN' field too.=0A ope= n FILE, "tar tRvf $file |" or die "$file: $!";=0A while (<FILE>) {=0A = if (m{^block (\d+): \S+ \d+/\d+\s+(\d+)\s.*/0*(\d+)$}) {=0A = my $offset =3D $1; # block offset in tar file=0A my $size = =3D $2; # size in bytes in tar file=0A my $filename...

...quot; - unless defined $tar && defined $file; - - die "$tar: file not found\n" - unless -f $tar; -} - -# Find the extent of the file within the tar file. -sub get_ready -{ - open (my $pipe, "-|", "tar", "--no-auto-compress", "-tRvf", $tar, $file) - or die "$tar: could not open or parse tar file, see errors above"; - while (<$pipe>) { - if (/^block\s(\d+):\s\S+\s\S+\s(\d+)/) { - # Add one for the tar header, and multiply by the block size. - $offset = ($1 + 1) * 512;...

...) ova ) @@ -135,6 +183,59 @@ object loop [dir] in + (* Find file in [tar] archive and return at which byte it starts and how + * long it is. + *) + let find_file_in_tar tar filename = + let lines = external_command (stringify_args [ "tar"; "tRvf"; tar ]) in + let rec loop lines = + match lines with + | [] -> raise Not_found + | line :: lines -> ( + (* Lines have the form: + * block <offset>: <perms> <owner>/<group> <size> <mdate> <mtime> <f...

v2: - 1/3: same as in previous version - 2/3: add check for libvirt version - 3/3: restore the disabled test The libvirt patch [2] that should fix our problem with 'raw' driver that is mentioned in [1] was merged. To realy fix things, there is at least one thing we have to change on our side. We have to be explicit about the driver in the JSON we generate. [1]

2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>: > Gandalf, is there an XVA file publically available (pref. not > too big) that we can look at? I can try to provide one, but it's simple: # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50 ---------- 0/0 42353 1970-01-01 01:00 ova.xml ---------- 0/0 1048576 1970-01-01 01:00 Ref:175/00000000

https://bugzilla.redhat.com/show_bug.cgi?id=1570407 This turned into quite an in-depth refactoring of how we handle OVAs. It also fixes a potential security issue. Rich.

...object > loop [dir] > in > > + (* Find file in [tar] archive and return at which byte it starts and how > + * long it is. > + *) > + let find_file_in_tar tar filename = > + let lines = external_command (stringify_args [ "tar"; "tRvf"; tar ]) in stringify_args is meant to be used for displaying, rather than passing it back as shell command. Please just build a command string as done in the rest of the places. > + let rec loop lines = > + match lines with > + | [] -> raise Not_found > +...

This is still a draft, not ready for commit yet. But feedback is welcomed. This series is related to the problem of inefficient import of OVA files. The needed enhancements of QEMU was merged into the codebase and should be available in QEMU 2.8. From there we can use 'size' and 'offset' options in raw driver to tell QEMU to use only subset of a file as an image. The first three