search for: guestfs_int_read_whole_file

Commits 3c27f3d91e1566854747bbe844186783fc84f3a8 and 1b6f0daa9ae7fcc94e389232d0c397816cda973d added an internal API for running commands asynchronously. It is only used by the copy-in and copy-out APIs. Unfortunately this made the command code very complex: it was almost impossible to redirect stderr to a file, and there were a lot of long-range dependencies through the file. It was also buggy:

On Tuesday, 12 September 2017 14:29:13 CEST Richard W.M. Jones wrote: > +/** > + * Generic functions for reading and writing the cache files, used > + * where we are just reading and writing plain text strings. > + */ > +static int > +generic_read_cache (guestfs_h *g, const char *filename, char **strp) > +{ > + if (access (filename, R_OK) == -1 && errno == ENOENT)

...+ * and qemu -devices ? output. + */ + if (access (qemu_help_filename, R_OK) == -1 || + access (qemu_devices_filename, R_OK) == -1) + goto do_test; + + debug (g, "loading previously cached test results"); + + data = safe_calloc (g, 1, sizeof *data); + + if (guestfs_int_read_whole_file (g, qemu_help_filename, + &data->qemu_help, NULL) == -1) { + guestfs_int_free_qemu_data (data); + return NULL; + } + + parse_qemu_version (g, data); + + if (guestfs_int_read_whole_file (g, qemu_devices_filename, +...

...- * and qemu -devices ? output. - */ - if (access (qemu_help_filename, R_OK) == -1 || - access (qemu_devices_filename, R_OK) == -1) - goto do_test; - - debug (g, "loading previously cached test results"); - - data = safe_calloc (g, 1, sizeof *data); - - if (guestfs_int_read_whole_file (g, qemu_help_filename, - &data->qemu_help, NULL) == -1) { - guestfs_int_free_qemu_data (data); - return NULL; - } - - parse_qemu_version (g, data->qemu_help, &data->qemu_version); - - if (guestfs_int_read_whole_file (g, qemu_d...

...t; + if (ret < 0) > + return NULL; > + > + tmpfile = safe_asprintf (g, "%s/filesystem_walk%d", g->tmpdir, ++g->unique); > + > + ret = guestfs_internal_filesystem_walk (g, mountable, tmpfile); > + if (ret < 0) > + return NULL; > + > + ret = guestfs_int_read_whole_file (g, tmpfile, &buf, &size); > + if (ret < 0) > + return NULL; > + > + return parse_filesystem_walk (g, buf, size); /* caller frees */ > +} > + > +/* Parse buf content and return dirents list. > + * Return a list of tsk_dirent on success, NULL on error. > +...

The patch I posted last week to disable mandatory locking for readonly drives (https://www.redhat.com/archives/libguestfs/2017-September/msg00013.html) was wrong in a couple of respects. Firstly it didn't work, which I didn't detect because my tests were testing the wrong thing. Oops. Secondly it used a simple version number check to detect qemu binaries implementing mandatory locking.

...s (filename, R_OK) == -1) { > if (errno == ENOENT) > return 0; /* no cache, run the test instead */ > perrorf (g, "access: %s", filename); > return -1; > } But isn't it OK since the following line will return the true error: if (guestfs_int_read_whole_file (g, filename, strp, NULL) == -1) ... ? > > +static int > > +generic_write_cache (guestfs_h *g, const char *filename, const char *str) > > +{ > > + CLEANUP_FCLOSE FILE *fp = fopen (filename, "w"); > > + if (fp == NULL) { > > + perrorf (g, "...

...t; + if (ret < 0) > + return NULL; > + > + tmpfile = safe_asprintf (g, "%s/filesystem_walk%d", g->tmpdir, ++g->unique); > + > + ret = guestfs_internal_filesystem_walk (g, mountable, tmpfile); > + if (ret < 0) > + return NULL; > + > + ret = guestfs_int_read_whole_file (g, tmpfile, &buf, &size); I think there should be checks for the size of the downloaded file, to avoid reading too much. > + if (ret < 0) > + return NULL; > + > + return parse_filesystem_walk (g, buf, size); /* caller frees */ > +} > + > +/* Parse buf conten...

Patches 1 & 2 were posted previously here: https://www.redhat.com/archives/libguestfs/2016-May/msg00134.html Patch 3 is a hack so I can test this using my own version of qemu (the `-L ?' stuff is not upstream). Patch 4 is where the real action takes place: Replace the caching of qemu features in blob-like files with a sqlite database. Probably the best way to approach this patch is to

v2 -> v3: - I addressed everything that Pino mentioned last time. - It's tricky to get a stable run when multiple copies of qemu are involved, because the same cache files get overwritten by parallel libguestfs. So I changed the names of the cache files to include the qemu binary key (size, mtime), which removes this conflict. This is in new patch 4/6. Rich.

Patches 1-4 are almost the same as they are when previously posted here: https://www.redhat.com/archives/libguestfs/2017-September/msg00039.html Patch 5 actually uses the mandatory locking test to turn off locking in the narrow case where a drive is opened readonly, and then only for the drive being inspected. Passes ordinary tests (‘check-direct’ and ‘check-valgrind-direct’). Rich.

...L; + + ret = guestfs_int_lazy_make_tmpdir (g); + if (ret < 0) + return NULL; + + tmpfile = safe_asprintf (g, "%s/filesystem_walk%d", g->tmpdir, ++g->unique); + + ret = guestfs_internal_filesystem_walk (g, mountable, tmpfile); + if (ret < 0) + return NULL; + + ret = guestfs_int_read_whole_file (g, tmpfile, &buf, &size); + if (ret < 0) + return NULL; + + return parse_filesystem_walk (g, buf, size); /* caller frees */ +} + +/* Parse buf content and return dirents list. + * Return a list of tsk_dirent on success, NULL on error. + */ +static struct guestfs_tsk_dirent_list *...

...> > + > > + tmpfile = safe_asprintf (g, "%s/filesystem_walk%d", g->tmpdir, > ++g->unique); > > + > > + ret = guestfs_internal_filesystem_walk (g, mountable, tmpfile); > > + if (ret < 0) > > + return NULL; > > + > > + ret = guestfs_int_read_whole_file (g, tmpfile, &buf, &size); > > + if (ret < 0) > > + return NULL; > > + > > + return parse_filesystem_walk (g, buf, size); /* caller frees */ > > +} > > + > > +/* Parse buf content and return dirents list. > > + * Return a list of tsk_...

...L; + + ret = guestfs_int_lazy_make_tmpdir (g); + if (ret < 0) + return NULL; + + tmpfile = safe_asprintf (g, "%s/filesystem_walk%d", g->tmpdir, ++g->unique); + + ret = guestfs_internal_filesystem_walk (g, mountable, tmpfile); + if (ret < 0) + return NULL; + + ret = guestfs_int_read_whole_file (g, tmpfile, &buf, &size); + if (ret < 0) + return NULL; + + return parse_filesystem_walk (g, buf, size); /* caller frees */ +} + +/* Parse buf content and return dirents list. + * Return a list of tsk_dirent on success, NULL on error. + */ +static struct guestfs_tsk_dirent_list *...

NOTE: Not for upstream, yet. This depends on 3 non-upstream patches. - The qemu rework in libguestfs, which in turn depends on what Pino is up to. - Support for '-L ?' in qemu: https://lists.gnu.org/archive/html/qemu-devel/2016-05/threads.html#02596 - Support for bios-fast.bin in qemu: https://lists.gnu.org/archive/html/qemu-devel/2016-05/threads.html#02408 This commit

This commit uses -bios bios-fast.bin if available, which basically stops SeaBIOS from trying to do PCI probing during boot, which is a waste of time when using the -kernel option. v1 -> v2: - Rebase on top of Pino's work. This still has 3 dependencies: - The qemu memoization work (v2). - Support for '-L ?' in qemu:

v2: - Increased the amount of collected information from the FS content. - Moved filesystem_walk0 as internal command. - Code improvement based on comments. - Adhere to project's coding style. - Better command documentation. - More robust tests. Patch ready for review, code available at: https://github.com/noxdafox/libguestfs/tree/filesystem_walk Matteo Cafasso (5): generator:

v3: - File size will be reported as - 1 if it cannot be retrieved. - Code improvements based on comments. Matteo Cafasso (5): generator: Added tsk_dirent struct configure: Added libtsk compile-time check daemon: Added internal_filesystem_walk command appliance: Added filesystem_walk command appliance: Added filesystem_walk command tests daemon/Makefile.am | 4 +-

The filesystem_walk API parses the FS internals of a partition and returns a list of all the files and directories contained within. It list deleted files and directories as well. For each node, it reports its relative path, its inode and its allocation status. This is the end user API for inspecting a disk partition content. The command can handle filenames with special characters. Example

v1 -> v2: - Rebase on top of Pino's version work. Two patches went upstream, these are the two remaining patches. Note the generation number is still inside the qemu.stat file. We could put it in the filename, I have no particular preference. Rich.