Libguestfs - Oct 2009 - [PATCH 1/3] Add guestfs_find0 API call which doesn't have limits

The current guestfs_find API call contains an undocumented limit: It
needs to marshall the whole list of filenames into a single protocol
message.

Unfortunately just about any Linux guest breaks this limit if you try
'guestfs_find ("/")' so this isn't much use.

These patches add a new API call (find0) which breaks this limit by
using a FileOut parameter.  It's slightly harder to use than
guestfs_find, but better suited to program cases (guestfs_find
remains, of course).

We also update virt-ls to use this in the 'virt-ls -R' case.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
-------------- next part --------------
>From aa3ddf71e9fe08ee246da7c2a6b570db031c6eaf Mon Sep 17 00:00:00 2001
From: Richard Jones <rjones at redhat.com>
Date: Mon, 19 Oct 2009 16:57:54 +0100
Subject: [PATCH 1/3] guestfs_find: Document protocol limits for this API call.

guestfs_find has to send the complete list of files in a single
protocol message (hence, limited to 2-4 MB).  Unfortunately on
a typical Linux guest, guestfs_find ("/") will exceed this limit
resulting in an error.

Therefore we should add an unlimited version of this call in a
future commit.  This commit just documents the current limit.
---
 src/generator.ml |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/src/generator.ml b/src/generator.ml
index 4f03c06..74890b4 100755
--- a/src/generator.ml
+++ b/src/generator.ml
@@ -2488,7 +2488,7 @@ C<resize2fs> sometimes gives an error about this and
sometimes not.
 In any case, it is always safe to call C<guestfs_e2fsck_f> before
 calling this function.");
 
-  ("find", (RStringList "names", [Pathname
"directory"]), 107, [],
+  ("find", (RStringList "names", [Pathname
"directory"]), 107, [ProtocolLimitWarning],
    [InitBasicFS, Always, TestOutputList (
       [["find"; "/"]], ["lost+found"]);
     InitBasicFS, Always, TestOutputList (
-- 
1.6.5.rc2

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
-------------- next part --------------
>From 8dce31f2c084914c6f1845e40ccbf26aec5a8278 Mon Sep 17 00:00:00 2001
From: Richard Jones <rjones at redhat.com>
Date: Mon, 19 Oct 2009 17:44:16 +0100
Subject: [PATCH 2/3] New API: find0 (unlimited version of find)

This adds a new API call guestfs_find0, which is like guestfs_find
but mainly doesn't suffer from the protocol limit of the earlier
command.  The earlier command is not deprecated because it is
still very useful.

guestfs_find0 uses a FileOut parameter and writes the results to
an external file.  The filenames in the output are separated by
ASCII NUL characters (so a bit like "find -print0").

There is also the addition of a regression test for this command.
---
 daemon/find.c             |   99 +++++++++++++++++++++++++++++++++++++++++++++
 regressions/Makefile.am   |    1 +
 regressions/test-find0.sh |   39 ++++++++++++++++++
 src/MAX_PROC_NR           |    2 +-
 src/generator.ml          |   37 ++++++++++++++++-
 5 files changed, 176 insertions(+), 2 deletions(-)
 create mode 100755 regressions/test-find0.sh

diff --git a/daemon/find.c b/daemon/find.c
index 2147c57..1cd3a7a 100644
--- a/daemon/find.c
+++ b/daemon/find.c
@@ -129,3 +129,102 @@ do_find (const char *dir)
 
   return res;			/* caller frees */
 }
+
+/* The code below assumes each path returned can fit into a protocol
+ * chunk.  If this turns out not to be true at some point in the
+ * future then we'll need to modify the code a bit to handle it, or
+ * make the chunk size a bit bigger.
+ */
+#if PATH_MAX > GUESTFS_MAX_CHUNK_SIZE
+#error "PATH_MAX > GUESTFS_MAX_CHUNK_SIZE"
+#endif
+
+/* Has one FileOut parameter. */
+int
+do_find0 (const char *dir)
+{
+  struct stat statbuf;
+  int r;
+  FILE *fp;
+  char *cmd;
+  char *sysrootdir;
+  size_t sysrootdirlen, len;
+  char str[GUESTFS_MAX_CHUNK_SIZE];
+
+  sysrootdir = sysroot_path (dir);
+  if (!sysrootdir) {
+    reply_with_perror ("malloc");
+    return -1;
+  }
+
+  r = stat (sysrootdir, &statbuf);
+  if (r == -1) {
+    reply_with_perror ("%s", dir);
+    free (sysrootdir);
+    return -1;
+  }
+  if (!S_ISDIR (statbuf.st_mode)) {
+    reply_with_error ("%s: not a directory", dir);
+    free (sysrootdir);
+    return -1;
+  }
+
+  sysrootdirlen = strlen (sysrootdir);
+
+  if (asprintf_nowarn (&cmd, "find %Q -print0", sysrootdir) ==
-1) {
+    reply_with_perror ("asprintf");
+    free (sysrootdir);
+    return -1;
+  }
+
+  if (verbose)
+    fprintf (stderr, "%s\n", cmd);
+
+  fp = popen (cmd, "r");
+  if (fp == NULL) {
+    reply_with_perror ("%s", cmd);
+    free (cmd);
+    return -1;
+  }
+  free (cmd);
+
+  /* Now we must send the reply message, before the file contents.  After
+   * this there is no opportunity in the protocol to send any error
+   * message back.  Instead we can only cancel the transfer.
+   */
+  reply (NULL, NULL);
+
+  while ((r = input_to_nul (fp, str, GUESTFS_MAX_CHUNK_SIZE)) > 0) {
+    if (verbose)
+      printf ("find0 string: %s\n", str);
+
+    len = strlen (str);
+    if (len <= sysrootdirlen)
+      continue;
+
+    /* Remove the directory part of the path before sending it. */
+    if (send_file_write (str + sysrootdirlen, r - sysrootdirlen) < 0) {
+      pclose (fp);
+      return -1;
+    }
+  }
+
+  if (ferror (fp)) {
+    perror (dir);
+    send_file_end (1);		/* Cancel. */
+    pclose (fp);
+    return -1;
+  }
+
+  if (pclose (fp) != 0) {
+    perror (dir);
+    send_file_end (1);		/* Cancel. */
+    return -1;
+  }
+
+  if (send_file_end (0))	/* Normal end of file. */
+    return -1;
+
+  return 0;
+}
+
diff --git a/regressions/Makefile.am b/regressions/Makefile.am
index 9112b94..058b60d 100644
--- a/regressions/Makefile.am
+++ b/regressions/Makefile.am
@@ -26,6 +26,7 @@ TESTS = \
 	rhbz503169c13.sh \
 	test-cancellation-download-librarycancels.sh \
 	test-cancellation-upload-daemoncancels.sh \
+	test-find0.sh \
 	test-noexec-stack.pl \
 	test-qemudie-killsub.sh \
 	test-qemudie-midcommand.sh \
diff --git a/regressions/test-find0.sh b/regressions/test-find0.sh
new file mode 100755
index 0000000..4849b64
--- /dev/null
+++ b/regressions/test-find0.sh
@@ -0,0 +1,39 @@
+#!/bin/sh -
+# libguestfs
+# Copyright (C) 2009 Red Hat Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+# Test find0 call.
+
+set -e
+
+rm -f test.out
+
+../fish/guestfish <<'EOF'
+add-ro ../images/test.iso
+run
+mount-ro /dev/sda /
+find0 / test.out
+EOF
+
+n=$(tr '\0' '\n' < test.out | grep '^known-[1-5]' |
wc -l)
+[ "$n" = 5 ] || {
+  echo find0: Invalid list of files
+  tr '\0' '\n' < test.out
+  exit 1
+}
+
+rm -f test.out
diff --git a/src/MAX_PROC_NR b/src/MAX_PROC_NR
index 6bb2f98..0f11735 100644
--- a/src/MAX_PROC_NR
+++ b/src/MAX_PROC_NR
@@ -1 +1 @@
-195
+196
diff --git a/src/generator.ml b/src/generator.ml
index 74890b4..cea5178 100755
--- a/src/generator.ml
+++ b/src/generator.ml
@@ -2525,7 +2525,9 @@ then the returned list from C<guestfs_find>
C</tmp> would be
 If C<directory> is not a directory, then this command returns
 an error.
 
-The returned list is sorted.");
+The returned list is sorted.
+
+See also C<guestfs_find0>.");
 
   ("e2fsck_f", (RErr, [Device "device"]), 108, [],
    [], (* lvresize tests this *)
@@ -3610,6 +3612,39 @@ You can use this command to test the connection through
to the daemon.
 
 See also C<guestfs_ping_daemon>.");
 
+  ("find0", (RErr, [Pathname "directory"; FileOut
"files"]), 196, [],
+   [], (* There is a regression test for this. *)
+   "find all files and directories, returning NUL-separated list",
+   "\
+This command lists out all files and directories, recursively,
+starting at C<directory>, placing the resulting list in the
+external file called C<files>.
+
+This command works the same way as C<guestfs_find> with the
+following exceptions:
+
+=over 4
+
+=item *
+
+The resulting list is written to an external file.
+
+=item *
+
+Items (filenames) in the result are separated
+by C<\\0> characters.  See L<find(1)> option I<-print0>.
+
+=item *
+
+This command is not limited in the number of names that it
+can return.
+
+=item *
+
+The result list is not sorted.
+
+=back");
+
 ]
 
 let all_functions = non_daemon_functions @ daemon_functions
-- 
1.6.5.rc2

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
-------------- next part --------------
>From 52b6f03decc2bf8d9cfc3642aa96474df3322930 Mon Sep 17 00:00:00 2001
From: Richard Jones <rjones at redhat.com>
Date: Mon, 19 Oct 2009 17:59:00 +0100
Subject: [PATCH 3/3] virt-ls: Modify to use find0 instead of find.

find0 is more scalable than find.  virt-ls will no longer
crash if asked to recursively list / on a Linux guest.
---
 tools/virt-ls |   16 ++++++++++++----
 1 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/tools/virt-ls b/tools/virt-ls
index 8f9c96f..2abfb0c 100755
--- a/tools/virt-ls
+++ b/tools/virt-ls
@@ -26,6 +26,7 @@ use Sys::Guestfs::Lib qw(open_guest get_partitions
resolve_windows_path
 use Pod::Usage;
 use Getopt::Long;
 use Locale::TextDomain 'libguestfs';
+use File::Temp qw/tempdir/;
 
 =encoding utf8
 
@@ -189,15 +190,22 @@ my $root_dev = $roots[0];
 my $os = $oses->{$root_dev};
 mount_operating_system ($g, $os);
 
-my @r;
 unless ($mode) {
-    @r = $g->ls ($directory);
+    my @r = $g->ls ($directory);
     print "$_\n" foreach @r;
 } elsif ($mode eq "l") {
     print ($g->ll ($directory));
 } else { # $mode eq "R"
-    @r = $g->find ($directory);
-    print "$_\n" foreach @r;
+    my $dir = tempdir (CLEANUP => 1);
+    $g->find0 ($directory, "$dir/find0");
+    open F, "$dir/find0" or die "$dir/find0: $!";
+    my $r;
+    my $line;
+    while (($r = read (F, $line, 1024)) > 0) {
+	$line =~ tr{\0}{\n};
+	print $line;
+    }
+    close F;
 }
 
 =head1 SEE ALSO
-- 
1.6.5.rc2

Missed out the diffstat:

 daemon/find.c             |   99 +++++++++++++++++++++++++++++++++++++++++++++
 regressions/Makefile.am   |    1 +
 regressions/test-find0.sh |   39 ++++++++++++++++++
 src/MAX_PROC_NR           |    2 +-
 src/generator.ml          |   39 +++++++++++++++++-
 tools/virt-ls             |   24 +++++-----
 6 files changed, 189 insertions(+), 15 deletions(-)


-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top