Richard W.M. Jones
2010-Jul-31 16:20 UTC
[Libguestfs] [PATCH] Enable coredumps to be captured from the appliance (RHBZ#619334).
https://bugzilla.redhat.com/show_bug.cgi?id=619334
This is a slightly unsatisfactory patch which allows coredumps to be
captured when they occur inside the appliance. You can capture
coredumps by doing:
export LIBGUESTFS_COREDUMP=/sysroot/core.%t.%p.%e
or equivalently:
g.set_coredump ("/sysroot/core.%t.%p.%e")
or variations thereof, see the manual page.
When a coredump occurs, a core file will be dropped into the root
directory of the currently mounted guest disk. If it is guestfsd
which crashes, then the appliance will exit, but I have checked that
the coredump is written and synched first. You can then copy this out
of the guest disk using guestfish and debug using gdb:
$ guestfish --ro -a guest_disk -m /dev/vg/lv_root
><fs> ll /
><fs> download /core.xxx.yyy.zzz core
$ gdb daemon/guestfsd core
http://fedoraproject.org/wiki/StackTraces#Obtaining_a_stack_trace_from_a_core_dump
Clearly you need to have a writable guest disk mounted for this to
work, and you have to not care too much about having large core files
written to it.
Ideally we would have liked to use a special capture disk, but it
turns out that the kernel contains code which disallows using a /dev
device (or any non-regular file) for coredumps:
http://lxr.linux.no/linux+v2.6.34.1/fs/exec.c#L1930
We could do it using a coredump-to-pipe, but I think it would be
better to just remove this limitation in the kernel.
Another shortcoming is that a coredump isn't a stack trace, and to get
a stack trace you really need a copy of the daemon around with
debugging enabled. It is possible we could change the RPM packaging
to ship this.
Luckily the suspected bug in Augeas which we are chasing (RHBZ#613967)
happens in aug_init or aug_save when the conditions above are
satisfied and we should be able to get a useful coredump. However
since I am unable to reproduce this bug at all, I cannot help with
this, so it's over to Marek and Matt.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
-------------- next part -------------->From 680d532bf92d1d3cbc208ad94d6133b69bcbb54a Mon Sep 17 00:00:00 2001
From: Richard Jones <rjones at redhat.com>
Date: Sat, 31 Jul 2010 14:35:32 +0100
Subject: [PATCH] Enable coredumps to be captured from the appliance
(RHBZ#619334).
---
daemon/guestfsd.c | 45 ++++++++++++++++++++++++++++++
fish/guestfish.pod | 6 ++++
src/generator.ml | 72 ++++++++++++++++++++++++++++++++++++++++++++++++
src/guestfs-internal.h | 1 +
src/guestfs.c | 29 +++++++++++++++++++
src/guestfs.pod | 6 ++++
src/launch.c | 8 +++++
7 files changed, 167 insertions(+), 0 deletions(-)
diff --git a/daemon/guestfsd.c b/daemon/guestfsd.c
index 49aca08..91d59cc 100644
--- a/daemon/guestfsd.c
+++ b/daemon/guestfsd.c
@@ -42,6 +42,8 @@
#include <arpa/inet.h>
#include <netinet/in.h>
#include <errno.h>
+#include <sys/time.h>
+#include <sys/resource.h>
#ifdef HAVE_PRINTF_H
# include <printf.h>
@@ -211,6 +213,49 @@ main (int argc, char *argv[])
}
#ifndef WIN32
+ /* Enable coredumps. */
+ if (cmdline) {
+ char *p = strstr (cmdline, "guestfs_coredump=");
+ if (p) {
+ p += 17;
+ size_t len = strcspn (p, " \t\n");
+ char *coredump = strndup (p, len);
+ if (!coredump) {
+ perror ("strndup");
+ exit (EXIT_FAILURE);
+ }
+
+#define CORE_PATTERN "/proc/sys/kernel/core_pattern"
+ int fd = open (CORE_PATTERN, O_WRONLY);
+ if (fd == -1) {
+ perror ("open: " CORE_PATTERN);
+ exit (EXIT_FAILURE);
+ }
+ if (write (fd, coredump, len) < (ssize_t) len) {
+ perror ("write: " CORE_PATTERN);
+ exit (EXIT_FAILURE);
+ }
+ if (close (fd) == -1) {
+ perror ("close: " CORE_PATTERN);
+ exit (EXIT_FAILURE);
+ }
+
+ struct rlimit limit = {
+ .rlim_cur = RLIM_INFINITY,
+ .rlim_max = RLIM_INFINITY
+ };
+ if (setrlimit (RLIMIT_CORE, &limit) == -1) {
+ perror ("setrlimit (RLIMIT_CORE)");
+ exit (EXIT_FAILURE);
+ }
+
+ if (verbose)
+ printf ("coredumps enabled to '%s'\n", coredump);
+ }
+ }
+#endif /* !WIN32 */
+
+#ifndef WIN32
/* Make sure SIGPIPE doesn't kill us. */
struct sigaction sa;
memset (&sa, 0, sizeof sa);
diff --git a/fish/guestfish.pod b/fish/guestfish.pod
index bfcec5c..9aa01df 100644
--- a/fish/guestfish.pod
+++ b/fish/guestfish.pod
@@ -875,6 +875,12 @@ home directory can be used. See L</FILES>.
Pass additional options to the guest kernel.
+=item LIBGUESTFS_COREDUMP
+
+Capture coredump(s) which happen inside the libguestfs appliance.
+For further details of this option, see
+L<guestfs(3)/guestfs_set_coredump>.
+
=item LIBGUESTFS_DEBUG
Set C<LIBGUESTFS_DEBUG=1> to enable verbose messages. This has the
diff --git a/src/generator.ml b/src/generator.ml
index 52e7aba..fea771a 100755
--- a/src/generator.ml
+++ b/src/generator.ml
@@ -940,6 +940,78 @@ to specify the QEMU interface emulation to use at run
time.");
This is the same as C<guestfs_add_drive_ro> but it allows you
to specify the QEMU interface emulation to use at run time.");
+ ("set_coredump", (RErr, [OptString "coredump"]), -1,
[FishAlias "coredump"],
+ [],
+ "capture coredump to file or block device",
+ "\
+This option can be used to debug coredumps in the libguestfs
+appliance, in the daemon or in auxiliary programs that it
+runs. Set this to a string which is written verbatim to
+C</proc/sys/kernel/core_pattern> in the appliance during launch.
+Set this to NULL to disable coredumps.
+
+Useful values for this setting include:
+
+=over 4
+
+=item C</sysroot/core.%t.%p.%e>
+
+Coredump(s) are written to the root directory of the mounted
+guest disk. The name of the core file includes the current
+time (seconds since the epoch), PID, and the name of the
+executable.
+
+Note that the prefix C</sysroot> is an implementation detail
+of the current appliance and may change in future.
+
+This requires that a guest disk is mounted and writable
+at the time that the coredump is generated, otherwise
+the coredump will be discarded.
+
+=back
+
+The following are I<not> useful values for this setting:
+
+=over 4
+
+=item C</dev/vdb>
+
+The Linux kernel explicitly disallows writing coredumps to
+non-regular files. Therefore using a coredump capture device
+like this does not work.
+
+=item C</tmp/core>
+
+=item C<filename>
+
+If you use paths or filenames like these, then coredump(s) are written
+to the root disk in the appliance. This is not very useful, because
+the root disk is a ramdisk which is forgotten when the appliance
+shuts down.
+
+=back
+
+The default is C<NULL> unless overridden by setting the
+C<LIBGUESTFS_COREDUMP> environment variable.
+
+Note that you must set this before calling C<guestfs_launch>,
+otherwise it has no effect.
+
+This debugging feature is not part of the ABI and may be
+changed or removed in a future release.");
+
+ ("get_coredump", (RConstOptString "coredump", []), -1,
[],
+ (* This cannot be tested with the current framework. The
+ * function can return NULL in normal operations, which the
+ * test framework interprets as an error.
+ *)
+ [],
+ "get the coredump debugging setting",
+ "\
+Return the coredump debugging setting.
+
+See C<guestfs_set_coredump>.");
+
]
(* daemon_functions are any functions which cause some action
diff --git a/src/guestfs-internal.h b/src/guestfs-internal.h
index 12ca0ec..9f90904 100644
--- a/src/guestfs-internal.h
+++ b/src/guestfs-internal.h
@@ -110,6 +110,7 @@ struct guestfs_h
char *path; /* Path to kernel, initrd. */
char *qemu; /* Qemu binary. */
char *append; /* Append to kernel command line. */
+ char *coredump; /* Coredump path. */
int memsize; /* Size of RAM (megabytes). */
diff --git a/src/guestfs.c b/src/guestfs.c
index c54462d..244f2c6 100644
--- a/src/guestfs.c
+++ b/src/guestfs.c
@@ -509,6 +509,35 @@ guestfs__get_append (guestfs_h *g)
}
int
+guestfs__set_coredump (guestfs_h *g, const char *coredump)
+{
+ if (coredump != NULL) {
+ /* Because we pass this in the kernel command line, it will
+ * currently fail if it contains any whitespace. We ought to fix
+ * this by escaping spaces and unescaping them in the appliance. XXX
+ */
+ size_t i;
+ for (i = 0; i < strlen (coredump); ++i)
+ if (c_isspace (coredump[i])) {
+ error (g, "coredump parameter must not contain whitespace");
+ return -1;
+ }
+ }
+
+ free (g->coredump);
+ g->coredump = NULL;
+
+ g->coredump = coredump ? safe_strdup (g, coredump) : NULL;
+ return 0;
+}
+
+const char *
+guestfs__get_coredump (guestfs_h *g)
+{
+ return g->coredump;
+}
+
+int
guestfs__set_memsize (guestfs_h *g, int memsize)
{
g->memsize = memsize;
diff --git a/src/guestfs.pod b/src/guestfs.pod
index 5a2e7a5..0749da7 100644
--- a/src/guestfs.pod
+++ b/src/guestfs.pod
@@ -1478,6 +1478,12 @@ time.
Pass additional options to the guest kernel.
+=item LIBGUESTFS_COREDUMP
+
+Capture coredump(s) which happen inside the libguestfs appliance.
+For further details of this option, see
+L</guestfs_set_coredump>.
+
=item LIBGUESTFS_DEBUG
Set C<LIBGUESTFS_DEBUG=1> to enable verbose messages. This
diff --git a/src/launch.c b/src/launch.c
index 0d7a3f3..5d3c236 100644
--- a/src/launch.c
+++ b/src/launch.c
@@ -598,6 +598,12 @@ guestfs__launch (guestfs_h *g)
add_cmdline (g, "-net");
add_cmdline (g, "nic,model=" NET_IF ",vlan=0");
+ /* Coredump parameter for Linux command line. */
+ size_t len = g->coredump ? 32 + strlen (g->coredump) : 0;
+ char coredump[len];
+ if (g->coredump)
+ snprintf (coredump, len, "guestfs_coredump=%s",
g->coredump);
+
#define LINUX_CMDLINE \
"panic=1 " /* force kernel to panic if daemon exits */ \
"console=ttyS0 " /* serial console */ \
@@ -613,11 +619,13 @@ guestfs__launch (guestfs_h *g)
"%s " /* (selinux) */
"%s " /* (vmchannel) */
"%s " /* (verbose) */
+ "%s " /* (coredump) */
"TERM=%s " /* (TERM environment variable) */
"%s", /* (append) */
g->selinux ? "selinux=1 enforcing=0" :
"selinux=0",
vmchannel ? vmchannel : "",
g->verbose ? "guestfs_verbose=1" : "",
+ g->coredump ? coredump : "",
getenv ("TERM") ? : "linux",
g->append ? g->append : "");
--
1.7.1
Richard W.M. Jones
2010-Jul-31 17:26 UTC
[Libguestfs] [PATCH] Enable coredumps to be captured from the appliance (RHBZ#619334).
On Sat, Jul 31, 2010 at 05:20:19PM +0100, Richard W.M. Jones wrote:> This is a slightly unsatisfactory patch which allows coredumps to be > captured when they occur inside the appliance. You can capture > coredumps by doing: > > export LIBGUESTFS_COREDUMP=/sysroot/core.%t.%p.%e > > or equivalently: > > g.set_coredump ("/sysroot/core.%t.%p.%e") > > or variations thereof, see the manual page.Actually it strikes me that this API is wrong. It should be something like: set-coredump-file eg. set-coredump-file /core.%t.%p.%e - Coredump to a mounted guest filesystem. This would add the /sysroot/ prefix automatically. The %-patterns would be some limited subset of what the kernel supports. set-coredump-device eg. set-coredump-device /dev/sdb - Coredump to a capture device. This would be implemented at first using a coredump-to-pipe, but we could later change the implemention to coredump directly to a device if we got a patch into the kernel to support it. clear-coredump - Clear the coredump file/device. Equivalent to setting it to NULL right now. This interface is more extensible and future-proof. We can extend it by adding more set-coredump-* methods. It's future-proof because it doesn't depend so closely on the current kernel implementation. The environment variable would have to be changed as well. - - - - The ideal future enhancement would be to get a stack trace without needing the coredump + gdb step. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
Richard W.M. Jones
2010-Aug-02 11:12 UTC
[Libguestfs] [PATCH v2] Enable coredumps to be captured from the appliance (RHBZ#619334).
The last version omitted to implement LIBGUESTFS_COREDUMP. This fixes that ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw -------------- next part -------------->From 927fce62f13bf2201bc93e2cba88334a383268e5 Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Sat, 31 Jul 2010 14:35:32 +0100 Subject: [PATCH] Enable coredumps to be captured from the appliance (RHBZ#619334). --- daemon/guestfsd.c | 45 ++++++++++++++++++++++++++++++ fish/guestfish.pod | 6 ++++ src/generator.ml | 72 ++++++++++++++++++++++++++++++++++++++++++++++++ src/guestfs-internal.h | 1 + src/guestfs.c | 36 ++++++++++++++++++++++++ src/guestfs.pod | 6 ++++ src/launch.c | 8 +++++ 7 files changed, 174 insertions(+), 0 deletions(-) diff --git a/daemon/guestfsd.c b/daemon/guestfsd.c index 49aca08..91d59cc 100644 --- a/daemon/guestfsd.c +++ b/daemon/guestfsd.c @@ -42,6 +42,8 @@ #include <arpa/inet.h> #include <netinet/in.h> #include <errno.h> +#include <sys/time.h> +#include <sys/resource.h> #ifdef HAVE_PRINTF_H # include <printf.h> @@ -211,6 +213,49 @@ main (int argc, char *argv[]) } #ifndef WIN32 + /* Enable coredumps. */ + if (cmdline) { + char *p = strstr (cmdline, "guestfs_coredump="); + if (p) { + p += 17; + size_t len = strcspn (p, " \t\n"); + char *coredump = strndup (p, len); + if (!coredump) { + perror ("strndup"); + exit (EXIT_FAILURE); + } + +#define CORE_PATTERN "/proc/sys/kernel/core_pattern" + int fd = open (CORE_PATTERN, O_WRONLY); + if (fd == -1) { + perror ("open: " CORE_PATTERN); + exit (EXIT_FAILURE); + } + if (write (fd, coredump, len) < (ssize_t) len) { + perror ("write: " CORE_PATTERN); + exit (EXIT_FAILURE); + } + if (close (fd) == -1) { + perror ("close: " CORE_PATTERN); + exit (EXIT_FAILURE); + } + + struct rlimit limit = { + .rlim_cur = RLIM_INFINITY, + .rlim_max = RLIM_INFINITY + }; + if (setrlimit (RLIMIT_CORE, &limit) == -1) { + perror ("setrlimit (RLIMIT_CORE)"); + exit (EXIT_FAILURE); + } + + if (verbose) + printf ("coredumps enabled to '%s'\n", coredump); + } + } +#endif /* !WIN32 */ + +#ifndef WIN32 /* Make sure SIGPIPE doesn't kill us. */ struct sigaction sa; memset (&sa, 0, sizeof sa); diff --git a/fish/guestfish.pod b/fish/guestfish.pod index bfcec5c..9aa01df 100644 --- a/fish/guestfish.pod +++ b/fish/guestfish.pod @@ -875,6 +875,12 @@ home directory can be used. See L</FILES>. Pass additional options to the guest kernel. +=item LIBGUESTFS_COREDUMP + +Capture coredump(s) which happen inside the libguestfs appliance. +For further details of this option, see +L<guestfs(3)/guestfs_set_coredump>. + =item LIBGUESTFS_DEBUG Set C<LIBGUESTFS_DEBUG=1> to enable verbose messages. This has the diff --git a/src/generator.ml b/src/generator.ml index 52e7aba..fea771a 100755 --- a/src/generator.ml +++ b/src/generator.ml @@ -940,6 +940,78 @@ to specify the QEMU interface emulation to use at run time."); This is the same as C<guestfs_add_drive_ro> but it allows you to specify the QEMU interface emulation to use at run time."); + ("set_coredump", (RErr, [OptString "coredump"]), -1, [FishAlias "coredump"], + [], + "capture coredump to file or block device", + "\ +This option can be used to debug coredumps in the libguestfs +appliance, in the daemon or in auxiliary programs that it +runs. Set this to a string which is written verbatim to +C</proc/sys/kernel/core_pattern> in the appliance during launch. +Set this to NULL to disable coredumps. + +Useful values for this setting include: + +=over 4 + +=item C</sysroot/core.%t.%p.%e> + +Coredump(s) are written to the root directory of the mounted +guest disk. The name of the core file includes the current +time (seconds since the epoch), PID, and the name of the +executable. + +Note that the prefix C</sysroot> is an implementation detail +of the current appliance and may change in future. + +This requires that a guest disk is mounted and writable +at the time that the coredump is generated, otherwise +the coredump will be discarded. + +=back + +The following are I<not> useful values for this setting: + +=over 4 + +=item C</dev/vdb> + +The Linux kernel explicitly disallows writing coredumps to +non-regular files. Therefore using a coredump capture device +like this does not work. + +=item C</tmp/core> + +=item C<filename> + +If you use paths or filenames like these, then coredump(s) are written +to the root disk in the appliance. This is not very useful, because +the root disk is a ramdisk which is forgotten when the appliance +shuts down. + +=back + +The default is C<NULL> unless overridden by setting the +C<LIBGUESTFS_COREDUMP> environment variable. + +Note that you must set this before calling C<guestfs_launch>, +otherwise it has no effect. + +This debugging feature is not part of the ABI and may be +changed or removed in a future release."); + + ("get_coredump", (RConstOptString "coredump", []), -1, [], + (* This cannot be tested with the current framework. The + * function can return NULL in normal operations, which the + * test framework interprets as an error. + *) + [], + "get the coredump debugging setting", + "\ +Return the coredump debugging setting. + +See C<guestfs_set_coredump>."); + ] (* daemon_functions are any functions which cause some action diff --git a/src/guestfs-internal.h b/src/guestfs-internal.h index 12ca0ec..9f90904 100644 --- a/src/guestfs-internal.h +++ b/src/guestfs-internal.h @@ -110,6 +110,7 @@ struct guestfs_h char *path; /* Path to kernel, initrd. */ char *qemu; /* Qemu binary. */ char *append; /* Append to kernel command line. */ + char *coredump; /* Coredump path. */ int memsize; /* Size of RAM (megabytes). */ diff --git a/src/guestfs.c b/src/guestfs.c index c54462d..10dd1e4 100644 --- a/src/guestfs.c +++ b/src/guestfs.c @@ -120,6 +120,12 @@ guestfs_create (void) if (!g->append) goto error; } + str = getenv ("LIBGUESTFS_COREDUMP"); + if (str) { + g->coredump = strdup (str); + if (!g->coredump) goto error; + } + /* Choose a suitable memory size. Previously we tried to choose * a minimal memory size, but this isn't really necessary since * recent QEMU and KVM don't do anything nasty like locking @@ -160,6 +166,7 @@ guestfs_create (void) free (g->path); free (g->qemu); free (g->append); + free (g->coredump); free (g); return NULL; } @@ -509,6 +516,35 @@ guestfs__get_append (guestfs_h *g) } int +guestfs__set_coredump (guestfs_h *g, const char *coredump) +{ + if (coredump != NULL) { + /* Because we pass this in the kernel command line, it will + * currently fail if it contains any whitespace. We ought to fix + * this by escaping spaces and unescaping them in the appliance. XXX + */ + size_t i; + for (i = 0; i < strlen (coredump); ++i) + if (c_isspace (coredump[i])) { + error (g, "coredump parameter must not contain whitespace"); + return -1; + } + } + + free (g->coredump); + g->coredump = NULL; + + g->coredump = coredump ? safe_strdup (g, coredump) : NULL; + return 0; +} + +const char * +guestfs__get_coredump (guestfs_h *g) +{ + return g->coredump; +} + +int guestfs__set_memsize (guestfs_h *g, int memsize) { g->memsize = memsize; diff --git a/src/guestfs.pod b/src/guestfs.pod index 5a2e7a5..0749da7 100644 --- a/src/guestfs.pod +++ b/src/guestfs.pod @@ -1478,6 +1478,12 @@ time. Pass additional options to the guest kernel. +=item LIBGUESTFS_COREDUMP + +Capture coredump(s) which happen inside the libguestfs appliance. +For further details of this option, see +L</guestfs_set_coredump>. + =item LIBGUESTFS_DEBUG Set C<LIBGUESTFS_DEBUG=1> to enable verbose messages. This diff --git a/src/launch.c b/src/launch.c index 0d7a3f3..5d3c236 100644 --- a/src/launch.c +++ b/src/launch.c @@ -598,6 +598,12 @@ guestfs__launch (guestfs_h *g) add_cmdline (g, "-net"); add_cmdline (g, "nic,model=" NET_IF ",vlan=0"); + /* Coredump parameter for Linux command line. */ + size_t len = g->coredump ? 32 + strlen (g->coredump) : 0; + char coredump[len]; + if (g->coredump) + snprintf (coredump, len, "guestfs_coredump=%s", g->coredump); + #define LINUX_CMDLINE \ "panic=1 " /* force kernel to panic if daemon exits */ \ "console=ttyS0 " /* serial console */ \ @@ -613,11 +619,13 @@ guestfs__launch (guestfs_h *g) "%s " /* (selinux) */ "%s " /* (vmchannel) */ "%s " /* (verbose) */ + "%s " /* (coredump) */ "TERM=%s " /* (TERM environment variable) */ "%s", /* (append) */ g->selinux ? "selinux=1 enforcing=0" : "selinux=0", vmchannel ? vmchannel : "", g->verbose ? "guestfs_verbose=1" : "", + g->coredump ? coredump : "", getenv ("TERM") ? : "linux", g->append ? g->append : ""); -- 1.7.1