From: Don Slutz <dslutz@verizon.com>
Subject was: Add xentrace/xen_crash
Changes from v1 to v2:
Rework to be closer to Xen "codeing standard".
Add a change to MAINTAINERS to list me as maintainer of xen-crashd.
Andrew Cooper:
Move out of xentrace, rename to xen-crashd.
Konrad Rzeszutek Wilk & David Vrabel:
Rework Copyright
Ian Campbell:
Add 1st pass on some documention on crash''s remote protocol.
Don Slutz (2):
xen-crashd: Connect crash with domain
MAINTAINERS: Add xen-crashd maintainer
.gitignore | 1 +
MAINTAINERS | 5 +
docs/misc/crash-remote.txt | 190 ++++++++++
tools/Makefile | 1 +
tools/xen-crashd/Makefile | 29 ++
tools/xen-crashd/xen-crashd.8 | 48 +++
tools/xen-crashd/xen-crashd.c | 833 ++++++++++++++++++++++++++++++++++++++++++
7 files changed, 1107 insertions(+)
create mode 100644 docs/misc/crash-remote.txt
create mode 100644 tools/xen-crashd/Makefile
create mode 100644 tools/xen-crashd/xen-crashd.8
create mode 100644 tools/xen-crashd/xen-crashd.c
--
1.8.4
From: Don Slutz <dslutz@verizon.com>
This allows crash to connect to a domU. Usage:
usage: /usr/lib/xen/bin/xen-crashd <domid> [<optional port>]
xen-crashd 1&
crash localhost:5001 /usr/lib/debug/lib/modules/3.8.11-100.fc17.x86_64/vmlinux
The domU will be paused while crash is connected. Currently the
code exits when crash disconnects.
Important: The domain running crash must be the same architecture as
the domU. Like both x86_64. Also the crash version must be new
enough to understand the domU''s kernel. If the kernel version are
different, additional arguments maybe need to get crash to work.
Best results will be had using the same version of linux while
running crash as the domU is running.
Signed-off-by: Don Slutz <dslutz@verizon.com>
---
.gitignore | 1 +
docs/misc/crash-remote.txt | 190 ++++++++++
tools/Makefile | 1 +
tools/xen-crashd/Makefile | 29 ++
tools/xen-crashd/xen-crashd.8 | 48 +++
tools/xen-crashd/xen-crashd.c | 833 ++++++++++++++++++++++++++++++++++++++++++
6 files changed, 1102 insertions(+)
create mode 100644 docs/misc/crash-remote.txt
create mode 100644 tools/xen-crashd/Makefile
create mode 100644 tools/xen-crashd/xen-crashd.8
create mode 100644 tools/xen-crashd/xen-crashd.c
diff --git a/.gitignore b/.gitignore
index 3253675..452209b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -279,6 +279,7 @@ tools/xentrace/xentrace_setsize
tools/xentrace/tbctl
tools/xentrace/xenctx
tools/xentrace/xentrace
+tools/xen-crashd/xen-crashd
tools/xm-test/ramdisk/buildroot
tools/xm-test/aclocal.m4
tools/xm-test/autom4te
diff --git a/docs/misc/crash-remote.txt b/docs/misc/crash-remote.txt
new file mode 100644
index 0000000..1b6baa5
--- /dev/null
+++ b/docs/misc/crash-remote.txt
@@ -0,0 +1,190 @@
+crash remote protocol
+ Written by Don Slutz <dslutz@verizon.com> Nov. 2013
+
+Introduction
+------------
+The utility crash supports looking at a domain using the tool
+xen-crashd. It does this by using the "crash remote protocol" which
+is described here.
+
+This protocol is over a TCP/IP connection. It is mostly text based
+with options seperated by a space. The one exception is that the
+data is return in raw following some text. At most 1 page of data
+is returned at one time. It is expected that a request will come as
+1 packet (which is not a valid TCP/IP statement, but works in 99.9%
+of the cases). A new mode (nil_mode) modifies the protocol to
+require a 0 byte on the end of all requests and responses. This is
+100% compatable with older versions, just allows newer code to do
+more then 1 recv() to get a request or response.
+
+Known commands:
+
+MACHINE_PID
+OPEN <filename>
+CLOSE <fid>
+READ_LIVE <fid> <address> <length>
[<vcpu>]
+PROC_VERSION
+PAGESIZE <mode>
+EXIT
+FIND_BOOTED_KERNEL
+READ_NETDUMP <address> <length>
+READ_MCLXCD <address> <length>
+READ <fid> <address> <length>
+TYPE <filename>
+LINUX_VERSION <filename>
+READ_GZIP <bufsize> <filename>
+DEBUGGING_SYMBOLS <filename>
+FIND_MODULE <release> <module>
+SUM <filename>
+MEMORY <type> <mode>
+MEMORY_DUMP <bufsize> <mode>
+NETDUMP_INIT <mfid> <dumpfile>
+LKCD_DUMP_INIT <mfid> <dumpfile>
+READ_LKCD <fid> <address> <length>
+S390_DUMP_INIT <mfid> <dumpfile>
+S390X_DUMP_INIT <mfid> <dumpfile>
+READ_S390D <fid> <address> <length>
+EXECUTE <bufsize> <mode>
+FETCH_LIVE_IP_SP_BP <vcpu>
+FETCH_LIVE_CR3 <vcpu>
+VTOP <vcpu> <address>
+NIL
+
+Most commands the return is an echo of the command but with the
+result(s) appended as text. If the command fails you return command
+with "<FAIL>". I.E. "OPEN <FAIL>" is a valid
return.
+
+Command notes:
+
+MACHINE_PID:
+
+ Return the <MACHINE_TYPE> that crash understands and
<server_pid>.
+
+OPEN:
+
+ Currently only /dev/mem, /dev/kmem, and /dev/vmem are supported. The return
is:
+ OPEN filename fid O_RDONLY filesize
+
+CLOSE:
+
+ Not currently supported.
+
+READ_LIVE:
+
+ This is the most complex. The optional <vcpu> argument will force
+ <address> to be a virtual one and will be translated to a physical
+ one. Otherwise a read from /dev/vmem will also select <address>
+ as virtual. For both /dev/mem and /dev/kmem <address> is a
+ physical one.
+
+ The return is also special in that it starts with either FAIL or
+ DONE, a space and a 7 digit error code (fail) or a 7 digit length
+ of data. Since the max size is one page the 7 digit number should
+ not overflow. For DONE, right after the 7 digit number the raw
+ data starts for <length> bytes. If any part of the data is not
+ mappable, the whole request fails.
+
+ The defined error code are:
+ 1: Transfer too big.
+ 2: Failed to map starting address.
+ 3: Failed to map ending address.
+ 4: Failed to convert virtual address to physical address.
+
+PROC_VERSION:
+
+ Return the value of "/proc/version". Has a special return where
+ "<FAIL>" is used for errors. Not currently supported.
+
+PAGESIZE:
+
+ Return the page size. Also returns info on server and number of
+ VCPUs (PAGESIZE <mode> <size> XEN <ncpu>).
+
+EXIT:
+
+ Close the connection. The return is also different in that it
+ will be "EXIT OK".
+
+
+Not currently supported.
+
+FIND_BOOTED_KERNEL:
+
+ Try to find the booted kernel. Returns the file name of the
+ booted vmlinux.
+
+READ_NETDUMP:
+
+
+READ_MCLXCD:
+
+
+READ:
+
+
+TYPE:
+
+
+LINUX_VERSION:
+
+
+READ_GZIP:
+
+
+DEBUGGING_SYMBOLS:
+
+
+FIND_MODULE:
+
+
+SUM:
+
+
+MEMORY:
+
+
+MEMORY_DUMP:
+
+
+NETDUMP_INIT:
+
+
+LKCD_DUMP_INIT:
+
+
+READ_LKCD:
+
+
+S390_DUMP_INIT:
+
+
+S390X_DUMP_INIT:
+
+
+READ_S390D:
+
+
+EXECUTE:
+
+
+The following are an extersion to crash.
+
+FETCH_LIVE_IP_SP_BP:
+
+ Return the current values of SP, IP, and BP. For x86 this is eip
+ or rip, etc. Also switches the default vcpu for READ_LIVE without
+ a <vcpu>.
+
+FETCH_LIVE_CR3:
+
+ Return the current ctrlreg[3] value. Also switches the default
+ vcpu for READ_LIVE without a <vcpu>.
+
+VTOP:
+
+ Return the convert of virtual address to physical address.
+ Physical page 0 means a convertion error.
+
+NIL:
+
+ Switch to NIL mode. I.E. require a - byte on the end.
diff --git a/tools/Makefile b/tools/Makefile
index 00c69ee..e0de80c 100644
--- a/tools/Makefile
+++ b/tools/Makefile
@@ -25,6 +25,7 @@ SUBDIRS-$(CONFIG_NetBSD) += xenbackendd
SUBDIRS-y += libfsimage
SUBDIRS-$(LIBXENAPI_BINDINGS) += libxen
SUBDIRS-$(CONFIG_Linux) += libvchan
+SUBDIRS-$(CONFIG_Linux) += xen-crashd
# do not recurse in to a dir we are about to delete
ifneq "$(MAKECMDGOALS)" "distclean"
diff --git a/tools/xen-crashd/Makefile b/tools/xen-crashd/Makefile
new file mode 100644
index 0000000..cca96fe
--- /dev/null
+++ b/tools/xen-crashd/Makefile
@@ -0,0 +1,29 @@
+XEN_ROOT=$(CURDIR)/../..
+include $(XEN_ROOT)/tools/Rules.mk
+
+CFLAGS += -Werror -g -O0
+
+CFLAGS += $(CFLAGS_libxenctrl)
+LDLIBS += $(LDLIBS_libxenctrl)
+
+LIBBIN = xen-crashd
+
+.PHONY: all
+all: build
+
+.PHONY: build
+build: $(BIN) $(LIBBIN)
+
+.PHONY: install
+install: build
+ [ -z "$(LIBBIN)" ] || $(INSTALL_DIR) $(DESTDIR)$(PRIVATE_BINDIR)
+ [ -z "$(LIBBIN)" ] || $(INSTALL_PROG) $(LIBBIN)
$(DESTDIR)$(PRIVATE_BINDIR)
+
+.PHONY: clean
+clean:
+ $(RM) *.a *.so *.o *.rpm $(BIN) $(LIBBIN) $(DEPS)
+
+xen-crashd: xen-crashd.o
+ $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS) $(APPEND_LDFLAGS)
+
+-include $(DEPS)
diff --git a/tools/xen-crashd/xen-crashd.8 b/tools/xen-crashd/xen-crashd.8
new file mode 100644
index 0000000..464f407
--- /dev/null
+++ b/tools/xen-crashd/xen-crashd.8
@@ -0,0 +1,48 @@
+.TH XEN-CRASHD 8 "15 November 2013" "Xen domain 0 utils"
+.SH NAME
+xen-crashd \- connect a Xen domain to
+.B crash
+.SH SYNOPSIS
+.B xen-crashd
+.I domain
+[
+.I PORT
+]
+.SH DESCRIPTION
+.B xen-crashd
+is used to connect
+.B crash
+to a \fIdomain\fR. The \fIdomain\fR will be paused while
+.B crash
+is connected. Currently the code exits when
+.B crash
+disconnects. The default
+.I PORT
+is 5001.
+
+.SH EXAMPLE
+ /usr/lib/xen/bin/xen-crashd
+.I 1
+&
+ \fBcrash\fR localhost:\fI5001\fR
/usr/lib/debug/lib/modules/3.8.11-100.fc17.x86_64/vmlinux
+
+.SH IMPORTANT
+The linux running
+.B crash
+must be the same architecture as the \fIdomain\fR. Like both
+x86_64. Also the
+.B crash
+version must be new enough to understand the \fIdomain\fR''s kernel.
+If the kernel version are different, additional arguments maybe need
+to get
+.B crash
+to work. Best results will be had using the same version of linux
+while running
+.B crash
+as the \fIdomain\fR is running.
+
+.SH AUTHOR
+Don Slutz <dslutz@verizon.com>
+
+.SH "SEE ALSO"
+crash(8)
diff --git a/tools/xen-crashd/xen-crashd.c b/tools/xen-crashd/xen-crashd.c
new file mode 100644
index 0000000..249bb0a
--- /dev/null
+++ b/tools/xen-crashd/xen-crashd.c
@@ -0,0 +1,833 @@
+/******************************************************************************
+ * tools/xen-crashd/xen-crashd.c
+ *
+ * Connect crash to DOMu.
+ *
+ * Copyright (C) 2012 by Cloud Switch, Inc.
+ * Copyright (C) 2013 by Terremark.
+ * Copyright (C) 2013 by Verizon.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <ctype.h>
+#include <time.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <sys/ioctl.h>
+#include <sys/time.h>
+#include <sys/stat.h>
+#include <netinet/in.h>
+#include <netdb.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <errno.h>
+#include <signal.h>
+#include <string.h>
+#include <inttypes.h>
+#include <getopt.h>
+
+#include "xenctrl.h"
+#include <xen/foreign/x86_32.h>
+#include <xen/foreign/x86_64.h>
+#include <xen/hvm/save.h>
+
+xc_interface *xc_handle = 0;
+int domid = 0;
+int debug = 0;
+
+#if defined (__i386__) || defined (__x86_64__)
+typedef unsigned long long guest_word_t;
+#define FMT_32B_WORD "%08llx"
+#define FMT_64B_WORD "%016llx"
+/* Word-length of the guest''s own data structures */
+int guest_word_size = sizeof (unsigned long);
+/* Word-length of the context record we get from xen */
+int ctxt_word_size = sizeof (unsigned long);
+int guest_protected_mode = 1;
+#elif defined(__arm__)
+#define NO_TRANSLATION
+typedef uint64_t guest_word_t;
+#define FMT_32B_WORD "%08llx"
+#define FMT_64B_WORD "%016llx"
+#elif defined(__aarch64__)
+#define NO_TRANSLATION
+typedef uint64_t guest_word_t;
+#define FMT_32B_WORD "%08lx"
+#define FMT_64B_WORD "%016lx"
+#endif
+
+#define STRNEQ(A, B) (B && \
+ (strncmp((char *)(A), (char *)(B), strlen((char *)(B))) == 0))
+#define FAILMSG "FAIL "
+#define DONEMSG "DONE "
+
+#define DATA_HDRSIZE (strlen("XXXX ") +
strlen("1234567") + 1)
+
+#define BUFSIZE 127
+#define READBUFSIZE DATA_HDRSIZE + XC_PAGE_SIZE
+
+#define MAX_REMOTE_FDS 10
+
+void
+print_now(void)
+{
+ struct timeval tp;
+ struct timezone tzp;
+ char *timeout;
+ int imil;
+
+ gettimeofday(&tp, &tzp);
+ timeout = ctime(&tp.tv_sec);
+ imil = tp.tv_usec / 1000;
+ timeout += 4; /* Skip day of week */
+ *(timeout + 3) = 0; /* Trim at space after month */
+ *(timeout + 6) = 0; /* Trim at space after day */
+ *(timeout + 15) = 0; /* Trim at seconds. */
+ *(timeout + 20) = 0; /* Trim after year. */
+ printf("%s %s %s %s.%.3d ", timeout + 4, timeout, timeout + 18,
+ timeout + 7, imil);
+}
+
+int
+tcp_read(int sock, const char *pv_buffer, size_t cb_buffer, int nil_mode)
+{
+ size_t cb_total = 0;
+
+ do
+ {
+ ssize_t cb_read = recv(sock, (void*)pv_buffer, cb_buffer,
MSG_NOSIGNAL);
+
+ if (cb_read <= 0)
+ return cb_read;
+ cb_total += cb_read;
+ if (!nil_mode && cb_total >= 4)
+ return cb_total;
+ if (!pv_buffer[cb_read - 1])
+ return cb_total;
+ cb_buffer -= cb_read;
+ pv_buffer = (char *)pv_buffer + cb_read;
+ } while (cb_buffer);
+
+ return cb_total;
+}
+
+int
+tcp_write(int sock, const void *pv_buffer, size_t cb_buffer)
+{
+ if (debug & 0x002) {
+ print_now();
+ printf("rtn: %s\n", (char*)pv_buffer);
+ }
+ do
+ {
+ size_t cb_now = cb_buffer;
+ ssize_t cb_written = send(sock, (const char *)pv_buffer, cb_now,
MSG_NOSIGNAL);
+
+ if (cb_written < 0)
+ return 1;
+ cb_buffer -= cb_written;
+ pv_buffer = (char *)pv_buffer + cb_written;
+ } while (cb_buffer);
+
+ return 0;
+}
+
+int
+tcp_write_string(int sock, const char *pv_buffer)
+{
+ if (debug & 0x020) {
+ print_now();
+ printf("rtn: %s\n", (char*)pv_buffer);
+ }
+ return tcp_write(sock, pv_buffer, strlen(pv_buffer) + 1);
+}
+
+static void *
+map_page(int vcpu, guest_word_t phys)
+{
+ static unsigned long previous_mfn = 0;
+ static void *mapped = NULL;
+
+ unsigned long mfn = phys >> XC_PAGE_SHIFT;
+ unsigned long offset = phys & ~XC_PAGE_MASK;
+
+ if (mapped && mfn == previous_mfn)
+ goto out;
+
+ if (mapped)
+ munmap(mapped, XC_PAGE_SIZE);
+
+ previous_mfn = mfn;
+
+ mapped = xc_map_foreign_range(xc_handle, domid, XC_PAGE_SIZE, PROT_READ,
mfn);
+
+ if (mapped == NULL) {
+ if (debug & 0x004) {
+ print_now();
+ printf("failed to map page for %08llx.\n", phys);
+ }
+ return NULL;
+ }
+
+ out:
+ return (void *)(mapped + offset);
+}
+
+static int
+copy_phys(int vcpu, char * pv_dst, size_t cb, guest_word_t phys)
+{
+ void * local_addr;
+ size_t cb_page = XC_PAGE_SIZE - (phys & ~XC_PAGE_MASK);
+
+ /* optimize for the case where access is completely within the first page.
*/
+ local_addr = map_page(vcpu, phys);
+ if (!local_addr) {
+ return 2;
+ }
+ if (cb <= cb_page) {
+ memcpy(pv_dst, local_addr, cb);
+ return 0;
+ }
+ memcpy(pv_dst, local_addr, cb_page);
+ pv_dst += cb_page;
+ phys += cb_page;
+ cb -= cb_page;
+
+ /* Max transfer is XC_PAGE_SIZE... */
+ if (cb > XC_PAGE_SIZE)
+ return 1;
+ local_addr = map_page(vcpu, phys);
+ if (!local_addr) {
+ return 3;
+ }
+ memcpy(pv_dst, local_addr, cb);
+ return 0;
+}
+
+static guest_word_t
+convert_to_phys(int vcpu, guest_word_t virt)
+{
+ unsigned long mfn = xc_translate_foreign_address(xc_handle, domid, vcpu,
virt);
+ unsigned long offset = virt & ~XC_PAGE_MASK;
+
+ return (mfn << XC_PAGE_SHIFT) + offset;
+}
+
+static int
+copy_virt(int vcpu, char * pv_dst, size_t cb, guest_word_t virt)
+{
+ void * local_addr;
+ unsigned long mfn = xc_translate_foreign_address(xc_handle, domid, vcpu,
virt);
+ unsigned long offset = virt & ~XC_PAGE_MASK;
+ guest_word_t phys = (mfn << XC_PAGE_SHIFT) + offset;
+ size_t cb_page = XC_PAGE_SIZE - offset;
+
+ if (mfn == 0)
+ return 4;
+ /* optimize for the case where access is completely within the first page.
*/
+
+ local_addr = map_page(vcpu, phys);
+ if (!local_addr) {
+ return 2;
+ }
+ if (cb <= cb_page) {
+ memcpy(pv_dst, local_addr, cb);
+ return 0;
+ }
+ memcpy(pv_dst, local_addr, cb_page);
+ pv_dst += cb_page;
+ phys += cb_page;
+ cb -= cb_page;
+
+ /* Max transfer is XC_PAGE_SIZE... */
+ if (cb > XC_PAGE_SIZE)
+ return 1;
+ local_addr = map_page(vcpu, phys);
+ if (!local_addr) {
+ return 3;
+ }
+ memcpy(pv_dst, local_addr, cb);
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ int port = 5001;
+ int sock;
+ unsigned int length;
+ struct sockaddr_in server;
+ int msgsock;
+ int nfds;
+ int reuseaddr;
+ int count;
+ int pass;
+ int i;
+ char recvbuf[BUFSIZE + 1];
+ char sendbuf[READBUFSIZE + 1];
+ int fds[MAX_REMOTE_FDS];
+ size_t cb_read = 0;
+
+ int ret;
+ int vcpu;
+ int nil_mode = 0;
+ vcpu_guest_context_any_t ctx;
+ xc_dominfo_t dominfo;
+ struct hvm_hw_cpu cpuctx;
+
+ if (argc < 2 || argc > 4) {
+ printf("usage: xen-crashd <domid> [<optional
port>]\n");
+ exit(-1);
+ }
+
+ domid = atoi(argv[1]);
+ if (domid==0) {
+ fprintf(stderr, "cannot trace dom0\n");
+ exit(-1);
+ }
+
+ if (argc > 2)
+ port = atoi(argv[2]);
+ if (argc > 3)
+ debug = strtol(argv[3], NULL, 0);
+ if (debug)
+ printf("xen-crashd: debug=%d(0x%x)\n", debug, debug);
+
+ signal(SIGPIPE, SIG_IGN);
+
+ sock = socket(AF_INET, SOCK_STREAM, 0);
+ if (sock < 0) {
+ perror("socket()");
+ exit(1);
+ }
+ reuseaddr = 1;
+ if (setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (char *)&reuseaddr,
+ sizeof reuseaddr) < 0) {
+ perror("setsockopt()");
+ exit(2);
+ }
+ server.sin_family = AF_INET;
+ server.sin_addr.s_addr = INADDR_ANY;
+ server.sin_port = htons(port);
+ count = -1;
+ errno = EADDRINUSE;
+ for (pass=0; (errno == EADDRINUSE) && (count < 0); pass++) {
+ if ((count = bind(sock,
+ (struct sockaddr *) & server,
+ sizeof server)) < 0) {
+ if (errno != EADDRINUSE) {
+ /* printf("Errno is %d\n", errno); */
+ perror("bind()");
+ exit(3);
+ }
+ sleep(1); /* Waiting for kernel... */
+ }
+ }
+ length = sizeof server;
+ if (getsockname(sock, (struct sockaddr *) & server, &length) <
0) {
+ perror("getsockname()");
+ exit(4);
+ }
+ print_now();
+ if (pass == 1)
+ printf("socket ready on port %d after 1 bind call\n", port);
+ else
+ printf("socket ready on port %d after %d bind calls\n", port,
pass);
+ listen(sock, 1);
+ msgsock = accept(sock, NULL, NULL);
+ if (msgsock == -1)
+ perror("accept()");
+ else {
+ print_now();
+ printf("Accepted a connection.\n");
+ close(sock); /* All done for now */
+ errno = 0; /* Just in case */
+ nfds = msgsock + 1;
+ }
+
+ xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
+ if (!xc_handle) {
+ perror("xc_interface_open");
+ exit(-1);
+ }
+
+ ret = xc_domain_getinfo(xc_handle, domid, 1, &dominfo);
+ if (ret < 0) {
+ perror("xc_domain_getinfo");
+ exit(-1);
+ }
+
+ ret = xc_domain_pause(xc_handle, domid);
+ if (ret < 0) {
+ perror("xc_domain_pause");
+ exit(-1);
+ }
+
+ vcpu = 0;
+ ret = xc_vcpu_getcontext(xc_handle, domid, vcpu, &ctx);
+ if (ret < 0) {
+ if (!dominfo.paused)
+ xc_domain_unpause(xc_handle, domid);
+ perror("xc_vcpu_getcontext");
+ exit(-1);
+ }
+
+ if (dominfo.hvm) {
+ xen_capabilities_info_t xen_caps = "";
+ if (xc_domain_hvm_getcontext_partial(
+ xc_handle, domid, HVM_SAVE_CODE(CPU),
+ vcpu, &cpuctx, sizeof cpuctx) != 0) {
+ perror("xc_domain_hvm_getcontext_partial");
+ exit(-1);
+ }
+ guest_word_size = (cpuctx.msr_efer & 0x400) ? 8 : 4;
+ guest_protected_mode = (cpuctx.cr0 & 0x1);
+ /* HVM guest context records are always host-sized */
+ if (xc_version(xc_handle, XENVER_capabilities, &xen_caps) != 0)
{
+ perror("xc_version");
+ exit(-1);
+ }
+ ctxt_word_size = (strstr(xen_caps, "xen-3.0-x86_64")) ? 8
: 4;
+ } else {
+ struct xen_domctl domctl;
+ memset(&domctl, 0, sizeof domctl);
+ domctl.domain = domid;
+ domctl.cmd = XEN_DOMCTL_get_address_size;
+ if (xc_domctl(xc_handle, &domctl) == 0)
+ ctxt_word_size = guest_word_size = domctl.u.address_size.size /
8;
+ }
+
+ for (i = 0; i < MAX_REMOTE_FDS; i++)
+ fds[i] = -1;
+
+ do {
+ cb_read = tcp_read(msgsock, recvbuf, BUFSIZE, nil_mode);
+ if (cb_read <= 0) {
+ close(msgsock);
+ msgsock = -1;
+ break;
+ }
+ recvbuf[cb_read] = 0;
+
+ if (debug & 0x001) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+
+ if (STRNEQ(recvbuf, "READ_LIVE"))
+ {
+ char *p1, *p2, *p3, *p4;
+ int rc2;
+ guest_word_t addr;
+ int fid;
+ int len;
+
+ p1 = strtok(recvbuf, " "); /* READ_LIVE */
+ p1 = strtok(NULL, " "); /* fid */
+ p2 = strtok(NULL, " "); /* paddress or vaddress */
+ p3 = strtok(NULL, " "); /* length */
+ p4 = strtok(NULL, " "); /* vaddress or vcpu */
+
+ fid = atoi(p1);
+ addr = strtoull(p2, NULL, 16);
+ len = atoi(p3);
+ if (len < 0 || len > XC_PAGE_SIZE)
+ {
+ print_now();
+ printf("bad len=%d page_size=%ld;%s %s %s %s %s\n",
+ len, XC_PAGE_SIZE, recvbuf, p1, p2, p3, p4);
+ len = 0;
+ rc2 = 4;
+ }
+
+ if (len)
+ {
+ if (p4) {
+ int my_cpu = atoi(p4);
+
+ if (debug & 0x004) {
+ guest_word_t p_addr = convert_to_phys(my_cpu, addr);
+ print_now();
+ printf("copy_virt(%d,,%d, 0x%llx)[%s %s %s %s]
p_addr=0x%lx\n",
+ my_cpu, len, addr, p1, p2, p3, p4,
(long)p_addr);
+ }
+ rc2 = copy_virt(my_cpu, &sendbuf[DATA_HDRSIZE], len,
addr);
+ } else if (fds[fid] == 3) {
+ if (debug & 0x004) {
+ guest_word_t p_addr = convert_to_phys(vcpu, addr);
+
+ print_now();
+ printf("copy_virt(%d,,%d, 0x%llx)[%s %s %s]
p_addr=0x%lx\n",
+ vcpu, len, addr, p1, p2, p3, (long)p_addr);
+ }
+ rc2 = copy_virt(vcpu, &sendbuf[DATA_HDRSIZE], len,
addr);
+ } else {
+ if (debug & 0x004) {
+ print_now();
+ printf("copy_phys(%d,,%d, 0x%llx)[%s %s
%s]\n",
+ vcpu, len, addr, p1, p2, p3);
+ }
+ rc2 = copy_phys(vcpu, &sendbuf[DATA_HDRSIZE], len,
addr);
+ }
+ if (rc2) {
+ if (debug & 0x004) {
+ print_now();
+ printf("Failed rc2=%d\n", rc2);
+ }
+ len = 0;
+ }
+ }
+
+ if (!len) {
+ snprintf(sendbuf, sizeof(sendbuf), "%s%07u", FAILMSG,
rc2);
+ } else {
+ snprintf(sendbuf, sizeof(sendbuf), "%s%07u", DONEMSG,
len);
+ }
+ if (tcp_write(msgsock, sendbuf, len + DATA_HDRSIZE))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "FETCH_LIVE_IP_SP_BP "))
+ {
+ char *p1, *p2;
+ int cpu;
+ long domu_ip = 0;
+ short domu_cs = 0;
+ short domu_ss = 0;
+ long domu_sp = 0;
+ long domu_bp = 0;
+
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ p1 = strtok(recvbuf, " "); /* FETCH_LIVE_IP_SP_BP */
+ p2 = strtok(NULL, " "); /* cpu */
+
+ cpu = atoi(p2);
+ if (cpu != vcpu) {
+ vcpu = cpu;
+ ret = xc_vcpu_getcontext(xc_handle, domid, vcpu, &ctx);
+ if (ret < 0) {
+ if (!dominfo.paused)
+ xc_domain_unpause(xc_handle, domid);
+ perror("xc_vcpu_getcontext");
+ exit(-1);
+ }
+ if (dominfo.hvm) {
+ if (xc_domain_hvm_getcontext_partial(
+ xc_handle, domid, HVM_SAVE_CODE(CPU),
+ vcpu, &cpuctx, sizeof cpuctx) != 0) {
+ perror("xc_domain_hvm_getcontext_partial");
+ exit(-1);
+ }
+ }
+ }
+
+ if (ctxt_word_size == 4) {
+ struct cpu_user_regs_x86_32 * regs = &(ctx.x32.user_regs);
+
+ domu_ip = regs->eip;
+ domu_sp = regs->esp;
+ domu_bp = regs->ebp;
+ domu_cs = regs->cs;
+ domu_ss = regs->ss;
+ } else {
+ struct cpu_user_regs_x86_64 * regs = &(ctx.x64.user_regs);
+
+ if (dominfo.hvm) {
+ domu_ip = cpuctx.rip;
+ domu_sp = cpuctx.rsp;
+ domu_bp = cpuctx.rbp;
+ domu_cs = cpuctx.cs_sel;
+ domu_ss = cpuctx.ss_sel;
+ if (debug & 0x100) {
+ if (domu_ip != regs->rip) {
+ printf("domu_ip(%lx) != rip(%lx)\n",
domu_ip, regs->rip);
+ }
+ if (domu_sp != regs->rsp) {
+ printf("domu_sp(%lx) != rsp(%lx)\n",
domu_sp, regs->rsp);
+ }
+ if (domu_bp != regs->rbp) {
+ printf("domu_bp(%lx) != rbp(%lx)\n",
domu_bp, regs->rbp);
+ }
+ if (domu_cs != regs->cs) {
+ printf("domu_cs(%x) != cs(%x)\n",
domu_cs, regs->cs);
+ }
+ if (domu_ss != regs->ss) {
+ printf("domu_ss(%x) != ss(%x)\n",
domu_ss, regs->ss);
+ }
+ }
+ } else {
+ domu_ip = regs->rip;
+ domu_sp = regs->rsp;
+ domu_bp = regs->rbp;
+ domu_cs = regs->cs;
+ domu_ss = regs->ss;
+ }
+ }
+
+ snprintf(sendbuf, sizeof(sendbuf), "%s %d %04x:%lx %04x:%lx
%lx",
+ p1, cpu, domu_cs, domu_ip, domu_ss, domu_sp, domu_bp);
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "FETCH_LIVE_CR3 "))
+ {
+ char *p1, *p2;
+ int cpu;
+ long domu_cr3 = 0;
+
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ p1 = strtok(recvbuf, " "); /* FETCH_LIVE_CR3 */
+ p2 = strtok(NULL, " "); /* cpu */
+
+ cpu = atoi(p2);
+ if (cpu != vcpu) {
+ vcpu = cpu;
+ ret = xc_vcpu_getcontext(xc_handle, domid, vcpu, &ctx);
+ if (ret < 0) {
+ if (!dominfo.paused)
+ xc_domain_unpause(xc_handle, domid);
+ perror("xc_vcpu_getcontext");
+ exit(-1);
+ }
+ if (dominfo.hvm) {
+ if (xc_domain_hvm_getcontext_partial(
+ xc_handle, domid, HVM_SAVE_CODE(CPU),
+ vcpu, &cpuctx, sizeof cpuctx) != 0) {
+ perror("xc_domain_hvm_getcontext_partial");
+ exit(-1);
+ }
+ }
+ }
+
+ if (ctxt_word_size == 4) {
+ domu_cr3 = ctx.x32.ctrlreg[3];
+ } else {
+ if (dominfo.hvm) {
+ domu_cr3 = cpuctx.cr3;
+ if (debug & 0x100) {
+ if (domu_cr3 != ctx.x64.ctrlreg[3]) {
+ printf("domu_cr3(%lx) != cr3(%lx)\n",
domu_cr3, ctx.x64.ctrlreg[3]);
+ }
+ }
+ } else {
+ domu_cr3 = ctx.x64.ctrlreg[3];
+ }
+ }
+
+ snprintf(sendbuf, sizeof(sendbuf), "%s %d %lx",
+ p1, cpu, domu_cr3);
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "MACHINE_PID"))
+ {
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+#if defined (__i386__) || defined (__x86_64__)
+ if (guest_word_size == 8)
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s %d",
+ recvbuf, "X86_64", 0);
+ else
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s %d",
+ recvbuf, "X86", 0);
+#elif defined(__arm__)
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s %d",
+ recvbuf, "ARM", 0);
+#elif defined(__aarch64__)
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s %d",
+ recvbuf, "ARM64", 0);
+#endif
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "VTOP"))
+ {
+ char *p1, *p2, *p3;
+ int cpu;
+ guest_word_t v_addr, p_addr;
+
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ p1 = strtok(recvbuf, " "); /* VTOP */
+ p2 = strtok(NULL, " "); /* cpu */
+ p3 = strtok(NULL, " "); /* vaddress */
+
+ cpu = atoi(p2);
+ v_addr = strtoull(p3, NULL, 16);
+
+ p_addr = convert_to_phys(cpu, v_addr);
+
+ snprintf(sendbuf, sizeof(sendbuf), "%s %d %lx %lx",
+ p1, cpu, (long)v_addr, (long)p_addr);
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "OPEN "))
+ {
+ char *p1;
+ char *file;
+
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ p1 = strtok(recvbuf, " "); /* OPEN */
+ file = strtok(NULL, " "); /* filename */
+
+ for (i = 0; i < MAX_REMOTE_FDS; i++) {
+ if (fds[i] == -1)
+ break;
+ }
+
+ if (i < MAX_REMOTE_FDS) {
+ if (STRNEQ(file, "/dev/mem"))
+ {
+ fds[i] = 1;
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s %d O_RDONLY
%lld", p1, file, i, 1024LL * 1024LL * 1024LL * 1024LL * 1024LL * 1024LL);
+ }
+ else if (STRNEQ(file, "/dev/kmem"))
+ {
+ fds[i] = 2;
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s %d O_RDONLY
%lld", p1, file, i, 1024LL * 1024LL * 1024LL * 1024LL * 1024LL * 1024LL);
+ }
+ else if (STRNEQ(file, "/dev/vmem"))
+ {
+ fds[i] = 3;
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s %d O_RDONLY
%lld", p1, file, i, 1024LL * 1024LL * 1024LL * 1024LL * 1024LL * 1024LL);
+ }
+ else
+ {
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s
<FAIL>", p1, file);
+ }
+ }
+ else
+ {
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s
<FAIL>", p1, file);
+ }
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "CLOSE "))
+ {
+ char *p1, *p2;
+
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ p1 = strtok(recvbuf, " "); /* SIZE */
+ p2 = strtok(NULL, " "); /* filename id */
+ snprintf(sendbuf, sizeof(sendbuf), "%s %s <FAIL>",
p1, p2);
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "PROC_VERSION"))
+ {
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ /*
+ * Perform the detection.
+ */
+ snprintf(sendbuf, sizeof(sendbuf), "<FAIL>");
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "PAGESIZE LIVE"))
+ {
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ snprintf(sendbuf, sizeof(sendbuf), "%s %ld XEN %d",
+ recvbuf, XC_PAGE_SIZE, dominfo.max_vcpu_id + 1);
+ if (tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ else if (STRNEQ(recvbuf, "EXIT"))
+ {
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ snprintf(sendbuf, sizeof(sendbuf), "%s OK", recvbuf);
+ tcp_write_string(msgsock, sendbuf);
+ break;
+ }
+ else if (STRNEQ(recvbuf, "NIL"))
+ {
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ nil_mode = 1;
+ snprintf(sendbuf, sizeof(sendbuf), "%s XEN OK", recvbuf);
+ tcp_write_string(msgsock, sendbuf);
+ break;
+ }
+ else
+ {
+ if (debug & 0x010) {
+ print_now();
+ printf("req: %s\n", recvbuf);
+ }
+ print_now();
+ printf("unknown: %s\n", recvbuf);
+ snprintf(sendbuf, sizeof(sendbuf), "%s <FAIL>",
recvbuf);
+ if(tcp_write_string(msgsock, sendbuf))
+ break;
+ }
+ } while (msgsock >= 0);
+ if (msgsock >= 0)
+ close(msgsock);
+
+ if (!dominfo.paused) {
+ ret = xc_domain_unpause(xc_handle, domid);
+ if (ret < 0) {
+ perror("xc_domain_unpause");
+ exit(-1);
+ }
+ }
+
+ xc_interface_close(xc_handle);
+ if (ret < 0) {
+ perror("xc_interface_close");
+ exit(-1);
+ }
+
+ return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
1.8.4
From: Don Slutz <dslutz@verizon.com> Signed-off-by: Don Slutz <dslutz@verizon.com> --- MAINTAINERS | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 33130e5..45227bf 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -345,6 +345,11 @@ F: xen/include/xsm/ F: xen/xsm/ F: docs/misc/xsm-flask.txt +XEN CRASHD +M: Don Slutz <dslutz@verizon.com +S: Supported +F: tools/xen-crashd/ + THE REST M: Keir Fraser <keir@xen.org> L: xen-devel@lists.xen.org -- 1.8.4
While testing the my new changes to crash (soon to be posted), I found
some bugs with v2 of the code. I have attached the v3 of patch #1. Also
a patch to convert v2 to v3.
-Don Slutz
On 11/15/13 14:20, Don Slutz wrote:> From: Don Slutz <dslutz@verizon.com>
>
> Subject was: Add xentrace/xen_crash
>
> Changes from v1 to v2:
> Rework to be closer to Xen "codeing standard".
> Add a change to MAINTAINERS to list me as maintainer of xen-crashd.
> Andrew Cooper:
> Move out of xentrace, rename to xen-crashd.
> Konrad Rzeszutek Wilk & David Vrabel:
> Rework Copyright
> Ian Campbell:
> Add 1st pass on some documention on crash''s remote protocol.
>
> Don Slutz (2):
> xen-crashd: Connect crash with domain
> MAINTAINERS: Add xen-crashd maintainer
>
> .gitignore | 1 +
> MAINTAINERS | 5 +
> docs/misc/crash-remote.txt | 190 ++++++++++
> tools/Makefile | 1 +
> tools/xen-crashd/Makefile | 29 ++
> tools/xen-crashd/xen-crashd.8 | 48 +++
> tools/xen-crashd/xen-crashd.c | 833
++++++++++++++++++++++++++++++++++++++++++
> 7 files changed, 1107 insertions(+)
> create mode 100644 docs/misc/crash-remote.txt
> create mode 100644 tools/xen-crashd/Makefile
> create mode 100644 tools/xen-crashd/xen-crashd.8
> create mode 100644 tools/xen-crashd/xen-crashd.c
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
On Fri, 2013-11-15 at 14:20 -0500, Don Slutz wrote:> Ian Campbell: > Add 1st pass on some documention on crash''s remote protocol.My concern with this was that we were using some sort of internal crash protocol which has no ABI stability guarantees etc. Documenting it in the Xen tree doesn''t really do anything to alleviate that concern. It should be a protocol which is published by the crash folks not us. Ideally they would agree to some sort of protocol stability level, or maybe you can show that the protocol had inbuilt backward and forward compatibility capabilities already? Even more concerning is [0] where one of the crash maintainers says:> It''s been deprecated for almost 10 years now. I don''t understand how > you have been able to even get it to build, never mind work as the mail > thread indicates?We surely don''t want to be adding code which relies on a protocol which has been deprecated for 10 years! Daniel K asked about gdbsx -- can that not speak to crash somehow? Or run on /proc/vmcore directly, or be extended to do so? Ian. [0] http://thread.gmane.org/gmane.linux.kernel.crash-dump.crash-utility/4714/focus=4736
On 11/29/13 05:26, Ian Campbell wrote:> On Fri, 2013-11-15 at 14:20 -0500, Don Slutz wrote: > >> Ian Campbell: >> Add 1st pass on some documention on crash''s remote protocol. > My concern with this was that we were using some sort of internal crash > protocol which has no ABI stability guarantees etc. Documenting it in > the Xen tree doesn''t really do anything to alleviate that concern. It > should be a protocol which is published by the crash folks not us.I have no issues with this. The only documentation I can find is: http://people.redhat.com/anderson/crash_whitepaper/> Ideally they would agree to some sort of protocol stability level, or > maybe you can show that the protocol had inbuilt backward and forward > compatibility capabilities already?It may not have the best backwards and forwards compatibility that could be designed. However so far I have been able to add features to a newer crash that have no issues with older "crashd" servers. And older crash code works fine with the newer "crashd" servers. This is not the 1st one of these I have coded, just the 1st that I can release.> Even more concerning is [0] where one of the crash maintainers says: >> It''s been deprecated for almost 10 years now. I don''t understand how >> you have been able to even get it to build, never mind work as the mail >> thread indicates? > We surely don''t want to be adding code which relies on a protocol which > has been deprecated for 10 years!The main reason that I know of is that crash in active mode (i.e. running live on machine A), is just so much simpler to use that using a remote crash on machine B talking to a crashd on machine A. This is because the crashd on machine A is in "live" mode. This means that slow or unresponsive systems cannot be examined using the remote protocol. And keeping the right kernel versions on machine B that you need is just overhead. With all this in mind, I was not surprised that it had been deprecated for 10 years. However with Xen in the mix, the machine A no longer needs to be active to run "crashd", in fact it can be paused, or running, or crashed, or shutdown, etc.> Daniel K asked about gdbsx -- can that not speak to crash somehow?It is clearly possible to write a remote crash to remote gdb server, but needing to run 2 servers to connect up crash is to me too complex. I could also embed the xen-crashd code in gdbsx by adding command line options. However very little code would be shared. Since I based xen-crashd off of xenctx, it currently uses libxc calls. gdbsx uses ioctl() directly to do the hyper calls. It does not appear to support physical addresses. It does not appear to support virtual address to physical address conversion. Quoteing from the crash whitepaper: Furthermore, to examine the contents of a live system''s kernel internals from user space, the only readily available option has been to use gdb on /proc/kcore. While gdb is an incredibly powerful tool, it is designed to debug user programs, and is not at all "kernel-aware". Consequently, using gdb alone has limited usefulness when looking at kernel memory, essentially constrained to the printing of kernel data structures */if/* the vmlinux file was built with the -g C flag, the disassembly of kernel text, and raw data dumps.> Or > run on /proc/vmcore directly, or be extended to do so?There is no /proc/vmcore in this case. Extending dom0 linux to provide /proc/1/vmcore, /proc/2/vmcore, etc. (I.E. /proc/<domid>/vmcore) would be a big change and designing a security model for these would also not be quick. Maybe this will help: [root@dcs-xen-54 tmp]# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2048 8 r----- 3928.9 P-1-0 1 3080 1 -b---- 18.0 [root@dcs-xen-54 tmp]# /usr/lib/xen/bin/xen-crashd 1& [1] 1447 [root@dcs-xen-54 tmp]# 2 Dec 13 11:38:01.042 socket ready on port 5001 after 1 bind call [root@dcs-xen-54 tmp]# crash --machdep phys_base=0x200000 localhost:5001 /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux crash 6.1.4 Copyright (C) 2002-2013 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. 2 Dec 13 11:38:08.917 Accepted a connection. WARNING: daemon cannot access /proc/version NOTE: setting phys_base to: 0x200000 GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux DUMPFILE: /dev/mem@localhost (remote live system) CPUS: 1 DATE: Mon Dec 2 11:37:02 2013 UPTIME: 00:33:11 LOAD AVERAGE: 0.01, 0.00, 0.00 TASKS: 81 NODENAME: P-1-0.TC5.CloudSwitch.com RELEASE: 2.6.18-128.el5 VERSION: #1 SMP Wed Jan 21 10:41:14 EST 2009 MACHINE: x86_64 (2400 Mhz) MEMORY: 3 GB PID: 0 COMMAND: "swapper" TASK: ffffffff802eeae0 [THREAD_INFO: ffffffff803dc000] CPU: 0 STATE: TASK_RUNNING (ACTIVE) crash> net NET_DEVICE NAME IP ADDRESS(ES) ffffffff80321e80 lo 127.0.0.1 ffff8100babd9000 eth1 172.16.64.65 ffff8100b6c96000 sit0 crash> q [1]+ Done /usr/lib/xen/bin/xen-crashd 1 Is almost the same as: [root@dcs-xen-54 tmp]# xl dump-core 1 p-1-0.vmore [root@dcs-xen-54 tmp]# crash p-1-0.vmore /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux crash 6.1.4 Copyright (C) 2002-2013 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux DUMPFILE: p-1-0.vmore CPUS: 1 DATE: Mon Dec 2 11:05:09 2013 UPTIME: 00:01:18 LOAD AVERAGE: 2.00, 0.70, 0.24 TASKS: 81 NODENAME: P-1-0.TC5.CloudSwitch.com RELEASE: 2.6.18-128.el5 VERSION: #1 SMP Wed Jan 21 10:41:14 EST 2009 MACHINE: x86_64 (2400 Mhz) MEMORY: 3 GB PANIC: "" PID: 0 COMMAND: "swapper" TASK: ffffffff802eeae0 [THREAD_INFO: ffffffff803dc000] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found crash> net NET_DEVICE NAME IP ADDRESS(ES) ffffffff80321e80 lo 127.0.0.1 ffff8100babd9000 eth1 172.16.64.65 ffff8100b6c96000 sit0 crash> quit With the changes in crash 7.0.4 (yet to be released), crash can be invoked in a remote "not live" mode, which is how it runs on a vmcore file. So if a DomU is paused, "xl dump-core;crash" and"xen-crashd;crash" will give the exact same answers in a lot less real time (xen-crashd case). -Don Slutz> > Ian. > > [0] > http://thread.gmane.org/gmane.linux.kernel.crash-dump.crash-utility/4714/focus=4736 >--------------010604090609060202070905 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 11/29/13 05:26, Ian Campbell wrote:<br> </div> <blockquote cite="mid:1385720807.20209.58.camel@kazak.uk.xensource.com" type="cite"> <pre wrap="">On Fri, 2013-11-15 at 14:20 -0500, Don Slutz wrote: </pre> <blockquote type="cite"> <pre wrap=""> Ian Campbell: Add 1st pass on some documention on crash''s remote protocol. </pre> </blockquote> <pre wrap="">My concern with this was that we were using some sort of internal crash protocol which has no ABI stability guarantees etc. Documenting it in the Xen tree doesn''t really do anything to alleviate that concern. It should be a protocol which is published by the crash folks not us.</pre> </blockquote> I have no issues with this. The only documentation I can find is:<br> <br> <blockquote> <a class="moz-txt-link-freetext" href="http://people.redhat.com/anderson/crash_whitepaper/">http://people.redhat.com/anderson/crash_whitepaper/</a><br> </blockquote> <br> <br> <blockquote cite="mid:1385720807.20209.58.camel@kazak.uk.xensource.com" type="cite"> <pre wrap="">Ideally they would agree to some sort of protocol stability level, or maybe you can show that the protocol had inbuilt backward and forward compatibility capabilities already? </pre> </blockquote> It may not have the best backwards and forwards compatibility that could be designed. However so far I have been able to add features to a newer crash that have no issues with older "crashd" servers. And older crash code works fine with the newer "crashd" servers. This is not the 1st one of these I have coded, just the 1st that I can release.<br> <blockquote cite="mid:1385720807.20209.58.camel@kazak.uk.xensource.com" type="cite"> <pre wrap="">Even more concerning is [0] where one of the crash maintainers says: </pre> <blockquote type="cite"> <pre wrap="">It''s been deprecated for almost 10 years now. I don''t understand how you have been able to even get it to build, never mind work as the mail thread indicates? </pre> </blockquote> <pre wrap="">We surely don''t want to be adding code which relies on a protocol which has been deprecated for 10 years! </pre> </blockquote> The main reason that I know of is that crash in active mode (i.e. running live on machine A), is just so much simpler to use that using a remote crash on machine B talking to a crashd on machine A. This is because the crashd on machine A is in "live" mode. This means that slow or unresponsive systems cannot be examined using the remote protocol. And keeping the right kernel versions on machine B that you need is just overhead.<br> <br> With all this in mind, I was not surprised that it had been deprecated for 10 years. However with Xen in the mix, the machine A no longer needs to be active to run "crashd", in fact it can be paused, or running, or crashed, or shutdown, etc.<br> <br> <br> <blockquote cite="mid:1385720807.20209.58.camel@kazak.uk.xensource.com" type="cite"> <pre wrap="">Daniel K asked about gdbsx -- can that not speak to crash somehow?</pre> </blockquote> It is clearly possible to write a remote crash to remote gdb server, but needing to run 2 servers to connect up crash is to me too complex. I could also embed the xen-crashd code in gdbsx by adding command line options. However very little code would be shared. Since I based xen-crashd off of xenctx, it currently uses libxc calls. gdbsx uses ioctl() directly to do the hyper calls. It does not appear to support physical addresses. It does not appear to support virtual address to physical address conversion. Quoteing from the crash whitepaper:<br> <blockquote>Furthermore, to examine the contents of a live system''s kernel internals from user space, the only readily available option has been to use <tt>gdb</tt> on <tt>/proc/kcore</tt>. While <tt>gdb</tt> is an incredibly powerful tool, it is designed to debug user programs, and is not at all "kernel-aware". Consequently, using <tt>gdb</tt> alone has limited usefulness when looking at kernel memory, essentially constrained to the printing of kernel data structures <b><i>if</i></b> the <tt>vmlinux</tt> file was built with the <tt>-g</tt> C flag, the disassembly of kernel text, and raw data dumps.</blockquote> <br> <blockquote cite="mid:1385720807.20209.58.camel@kazak.uk.xensource.com" type="cite"> <pre wrap=""> Or run on /proc/vmcore directly, or be extended to do so?</pre> </blockquote> There is no /proc/vmcore in this case. Extending dom0 linux to provide /proc/1/vmcore, /proc/2/vmcore, etc. (I.E. /proc/<domid>/vmcore) would be a big change and designing a security model for these would also not be quick.<br> <br> Maybe this will help:<br> <br> <tt>[root@dcs-xen-54 tmp]# xl list</tt><tt><br> </tt><tt>Name ID Mem VCPUs State Time(s)</tt><tt><br> </tt><tt>Domain-0 0 2048 8 r----- 3928.9</tt><tt><br> </tt><tt>P-1-0 1 3080 1 -b---- 18.0</tt><tt><br> </tt><br> <br> <blockquote><tt>[root@dcs-xen-54 tmp]# /usr/lib/xen/bin/xen-crashd 1&</tt><tt><br> </tt><tt>[1] 1447</tt><tt><br> </tt><tt>[root@dcs-xen-54 tmp]# 2 Dec 13 11:38:01.042 socket ready on port 5001 after 1 bind call</tt><tt><br> </tt><tt><br> </tt><tt>[root@dcs-xen-54 tmp]# crash --machdep phys_base=0x200000 localhost:5001 /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux</tt><tt><br> </tt><tt><br> </tt><tt>crash 6.1.4</tt><tt><br> </tt><tt>Copyright (C) 2002-2013 Red Hat, Inc.</tt><tt><br> </tt><tt>Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation</tt><tt><br> </tt><tt>Copyright (C) 1999-2006 Hewlett-Packard Co</tt><tt><br> </tt><tt>Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited</tt><tt><br> </tt><tt>Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.</tt><tt><br> </tt><tt>Copyright (C) 2005, 2011 NEC Corporation</tt><tt><br> </tt><tt>Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.</tt><tt><br> </tt><tt>Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.</tt><tt><br> </tt><tt>This program is free software, covered by the GNU General Public License,</tt><tt><br> </tt><tt>and you are welcome to change it and/or distribute copies of it under</tt><tt><br> </tt><tt>certain conditions. Enter "help copying" to see the conditions.</tt><tt><br> </tt><tt>This program has absolutely no warranty. Enter "help warranty" for details.</tt><tt><br> </tt><tt> </tt><tt><br> </tt><tt> 2 Dec 13 11:38:08.917 Accepted a connection.</tt><tt><br> </tt><tt>WARNING: daemon cannot access /proc/version</tt><tt><br> </tt><tt><br> </tt><tt>NOTE: setting phys_base to: 0x200000</tt><tt><br> </tt><tt><br> </tt><tt>GNU gdb (GDB) 7.3.1</tt><tt><br> </tt><tt>Copyright (C) 2011 Free Software Foundation, Inc.</tt><tt><br> </tt><tt>License GPLv3+: GNU GPL version 3 or later <a class="moz-txt-link-rfc2396E" href="http://gnu.org/licenses/gpl.html"><http://gnu.org/licenses/gpl.html></a></tt><tt><br> </tt><tt>This is free software: you are free to change and redistribute it.</tt><tt><br> </tt><tt>There is NO WARRANTY, to the extent permitted by law. Type "show copying"</tt><tt><br> </tt><tt>and "show warranty" for details.</tt><tt><br> </tt><tt>This GDB was configured as "x86_64-unknown-linux-gnu"...</tt><tt><br> </tt><tt><br> </tt><tt> KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux</tt><tt><br> </tt><tt> DUMPFILE: /dev/mem@localhost (remote live system)</tt><tt><br> </tt><tt> CPUS: 1</tt><tt><br> </tt><tt> DATE: Mon Dec 2 11:37:02 2013</tt><tt><br> </tt><tt> UPTIME: 00:33:11</tt><tt><br> </tt><tt>LOAD AVERAGE: 0.01, 0.00, 0.00</tt><tt><br> </tt><tt> TASKS: 81</tt><tt><br> </tt><tt> NODENAME: P-1-0.TC5.CloudSwitch.com</tt><tt><br> </tt><tt> RELEASE: 2.6.18-128.el5</tt><tt><br> </tt><tt> VERSION: #1 SMP Wed Jan 21 10:41:14 EST 2009</tt><tt><br> </tt><tt> MACHINE: x86_64 (2400 Mhz)</tt><tt><br> </tt><tt> MEMORY: 3 GB</tt><tt><br> </tt><tt> PID: 0</tt><tt><br> </tt><tt> COMMAND: "swapper"</tt><tt><br> </tt><tt> TASK: ffffffff802eeae0 [THREAD_INFO: ffffffff803dc000]</tt><tt><br> </tt><tt> CPU: 0</tt><tt><br> </tt><tt> STATE: TASK_RUNNING (ACTIVE)</tt><tt><br> </tt><tt><br> </tt><tt>crash> net<br> NET_DEVICE NAME IP ADDRESS(ES)<br> ffffffff80321e80 lo 127.0.0.1<br> ffff8100babd9000 eth1 172.16.64.65<br> ffff8100b6c96000 sit0 <br> crash> q</tt><tt><br> </tt><tt>[1]+ Done /usr/lib/xen/bin/xen-crashd 1</tt><tt><br> </tt></blockquote> <br> Is almost the same as:<br> <br> <blockquote><tt>[root@dcs-xen-54 tmp]# xl dump-core 1 p-1-0.vmore</tt><tt><br> </tt><tt>[root@dcs-xen-54 tmp]# crash p-1-0.vmore /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux </tt><tt><br> </tt><tt><br> </tt><tt>crash 6.1.4</tt><tt><br> </tt><tt>Copyright (C) 2002-2013 Red Hat, Inc.</tt><tt><br> </tt><tt>Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation</tt><tt><br> </tt><tt>Copyright (C) 1999-2006 Hewlett-Packard Co</tt><tt><br> </tt><tt>Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited</tt><tt><br> </tt><tt>Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.</tt><tt><br> </tt><tt>Copyright (C) 2005, 2011 NEC Corporation</tt><tt><br> </tt><tt>Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.</tt><tt><br> </tt><tt>Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.</tt><tt><br> </tt><tt>This program is free software, covered by the GNU General Public License,</tt><tt><br> </tt><tt>and you are welcome to change it and/or distribute copies of it under</tt><tt><br> </tt><tt>certain conditions. Enter "help copying" to see the conditions.</tt><tt><br> </tt><tt>This program has absolutely no warranty. Enter "help warranty" for details.</tt><tt><br> </tt><tt> </tt><tt><br> </tt><tt>GNU gdb (GDB) 7.3.1</tt><tt><br> </tt><tt>Copyright (C) 2011 Free Software Foundation, Inc.</tt><tt><br> </tt><tt>License GPLv3+: GNU GPL version 3 or later <a class="moz-txt-link-rfc2396E" href="http://gnu.org/licenses/gpl.html"><http://gnu.org/licenses/gpl.html></a></tt><tt><br> </tt><tt>This is free software: you are free to change and redistribute it.</tt><tt><br> </tt><tt>There is NO WARRANTY, to the extent permitted by law. Type "show copying"</tt><tt><br> </tt><tt>and "show warranty" for details.</tt><tt><br> </tt><tt>This GDB was configured as "x86_64-unknown-linux-gnu"...</tt><tt><br> </tt><tt><br> </tt><tt> KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux</tt><tt><br> </tt><tt> DUMPFILE: p-1-0.vmore</tt><tt><br> </tt><tt> CPUS: 1</tt><tt><br> </tt><tt> DATE: Mon Dec 2 11:05:09 2013</tt><tt><br> </tt><tt> UPTIME: 00:01:18</tt><tt><br> </tt><tt>LOAD AVERAGE: 2.00, 0.70, 0.24</tt><tt><br> </tt><tt> TASKS: 81</tt><tt><br> </tt><tt> NODENAME: P-1-0.TC5.CloudSwitch.com</tt><tt><br> </tt><tt> RELEASE: 2.6.18-128.el5</tt><tt><br> </tt><tt> VERSION: #1 SMP Wed Jan 21 10:41:14 EST 2009</tt><tt><br> </tt><tt> MACHINE: x86_64 (2400 Mhz)</tt><tt><br> </tt><tt> MEMORY: 3 GB</tt><tt><br> </tt><tt> PANIC: ""</tt><tt><br> </tt><tt> PID: 0</tt><tt><br> </tt><tt> COMMAND: "swapper"</tt><tt><br> </tt><tt> TASK: ffffffff802eeae0 [THREAD_INFO: ffffffff803dc000]</tt><tt><br> </tt><tt> CPU: 0</tt><tt><br> </tt><tt> STATE: TASK_RUNNING (ACTIVE)</tt><tt><br> </tt><tt> WARNING: panic task not found</tt><tt><br> </tt><tt><br> </tt><tt>crash> net<br> NET_DEVICE NAME IP ADDRESS(ES)<br> ffffffff80321e80 lo 127.0.0.1<br> ffff8100babd9000 eth1 172.16.64.65<br> ffff8100b6c96000 sit0 <br> crash> quit </tt><tt><br> </tt></blockquote> With the changes in crash 7.0.4 (yet to be released), crash can be invoked in a remote "not live" mode, which is how it runs on a vmcore file.<br> <br> So if a DomU is paused, "<tt>xl dump-core;crash" </tt>and<tt> "xen-crashd;crash"</tt> will give the exact same answers in a lot less real time (xen-crashd case).<br> <br> <br> -Don Slutz<br> <blockquote cite="mid:1385720807.20209.58.camel@kazak.uk.xensource.com" type="cite"> <pre wrap=""> Ian. [0] <a class="moz-txt-link-freetext" href="http://thread.gmane.org/gmane.linux.kernel.crash-dump.crash-utility/4714/focus=4736">http://thread.gmane.org/gmane.linux.kernel.crash-dump.crash-utility/4714/focus=4736</a> </pre> </blockquote> <br> </body> </html> --------------010604090609060202070905-- --===============8441855469695681043=Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============8441855469695681043==--
On Fri, Nov 29, 2013 at 10:26 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Fri, 2013-11-15 at 14:20 -0500, Don Slutz wrote: > >> Ian Campbell: >> Add 1st pass on some documention on crash''s remote protocol. > > My concern with this was that we were using some sort of internal crash > protocol which has no ABI stability guarantees etc. Documenting it in > the Xen tree doesn''t really do anything to alleviate that concern. It > should be a protocol which is published by the crash folks not us. > Ideally they would agree to some sort of protocol stability level, or > maybe you can show that the protocol had inbuilt backward and forward > compatibility capabilities already? > > Even more concerning is [0] where one of the crash maintainers says: >> It''s been deprecated for almost 10 years now. I don''t understand how >> you have been able to even get it to build, never mind work as the mail >> thread indicates? > > We surely don''t want to be adding code which relies on a protocol which > has been deprecated for 10 years! > > Daniel K asked about gdbsx -- can that not speak to crash somehow? Or > run on /proc/vmcore directly, or be extended to do so?Based on the interface concerns, I''m taking it you think this is probably not 4.4 material at this point? -George