Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH v3 00/18] libxl: domain save/restore: run in a separate process
This is v4 of my series to asyncify save/restore. All comments have
been addressed.
I have retested this with HVM (RHEL) and PV guests, for both localhost
migration and save/restore. RHEL seems to have some unreliability
which occurs with the baseline too. I was not able to test stub dms
at all because when trying to boot my RHEL guest with a stub dm the dm
domain immediately crashed. (I discovered that my previous tests had
occasionally not been using all of the new code.)
In the list below "A" indicates a patch which has been acked
sufficiently to go in (assuming its dependencies were to go in too).
"*" indicates a new patch in v4 and "+" indicates a patch
with
nontrivial changes from v3.
Preparatory fixes:
A libxc: xc_domain_restore, make toolstack_restore const-correct
* libxc: Do not segfault if (e.g.) switch_qemu_logdirty fails
Preparatory reshuffling:
A libxl: domain save: rename variables etc.
A libxl: domain restore: reshuffle, preparing for ao
+ libxl: domain save: API changes for asynchrony
The meat:
+ libxl: domain save/restore: run in a separate process
Some more fixups:
A libxl: rename libxl_dom:save_helper to physmap_path
A libxl: provide libxl__xs_*_checked and libxl__xs_transaction_*
+ A libxl: wait for qemu to acknowledge logdirty command
Asyncify writing of qemu save file, too:
A libxl: datacopier: provide "prefix data" facility
A libxl: prepare for asynchronous writing of qemu save file
A libxl: Make libxl__domain_save_device_model asynchronous
Fix gc_opt handling:
A libxl: Add a gc to libxl_get_cpu_topology
A libxl: Do not pass NULL as gc_opt; introduce NOGC
A libxl: Get compiler to warn about gc_opt==NULL
Work on essentially-unrelated bugs:
A xl: Handle return value from libxl_domain_suspend correctly
A libxl: do not leak dms->saved_state
A libxl: do not leak spawned middle children
A libxl: do not leak an event struct on ignored ao progress
libxl: further fixups re LIBXL_DOMAIN_TYPE
libxl: DO NOT APPLY enforce prohibition on internal
Thanks,
Ian.
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 01/21] libxc: xc_domain_restore, make toolstack_restore const-correct
Update the one provider of this callback, in libxl.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in v3:
* No longer introduce function pointer typedefs into the libxc API.
---
tools/libxc/xenguest.h | 2 +-
tools/libxl/libxl_dom.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/libxc/xenguest.h b/tools/libxc/xenguest.h
index 91d53f7..707e31c 100644
--- a/tools/libxc/xenguest.h
+++ b/tools/libxc/xenguest.h
@@ -92,7 +92,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
uint32_t max_iter
/* callbacks provided by xc_domain_restore */
struct restore_callbacks {
/* callback to restore toolstack specific data */
- int (*toolstack_restore)(uint32_t domid, uint8_t *buf,
+ int (*toolstack_restore)(uint32_t domid, const uint8_t *buf,
uint32_t size, void* data);
/* to be provided as the last argument to each callback function */
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 10f8c1f..677db1d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -467,13 +467,13 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
domid, phys_offset, node);
}
-static int libxl__toolstack_restore(uint32_t domid, uint8_t *buf,
+static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data)
{
libxl__gc *gc = (libxl__gc *) data;
libxl_ctx *ctx = gc->owner;
int i, ret;
- uint8_t *ptr = buf;
+ const uint8_t *ptr = buf;
uint32_t count = 0, version = 0;
struct libxl__physmap_info* pi;
char *xs_path;
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 02/21] libxc: Do not segfault if (e.g.) switch_qemu_logdirty fails
In xc_domain_save the local variable `ob'' is initialised to NULL.
There are then various startup actions. Some of these `goto out'' on
failure; for example the call to callbacks->switch_qemu_logdirty on
l.978. However, out is used both by success and error paths. So it
attempts (l.2043) to flush the current output buffer. If ob has not
yet been assigned a non-NULL value, this segfaults. So make the call
to outbuf_flush conditional on ob.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tools/libxc/xc_domain_save.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/tools/libxc/xc_domain_save.c b/tools/libxc/xc_domain_save.c
index fcc7718..c359649 100644
--- a/tools/libxc/xc_domain_save.c
+++ b/tools/libxc/xc_domain_save.c
@@ -2040,7 +2040,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t
dom, uint32_t max_iter
}
/* Flush last write and discard cache for file. */
- if ( outbuf_flush(xch, ob, io_fd) < 0 ) {
+ if ( ob && outbuf_flush(xch, ob, io_fd) < 0 ) {
PERROR("Error when flushing output buffer");
rc = 1;
}
--
1.7.2.5
Preparatory work for making domain suspend asynchronous:
* Rename `struct suspendinfo'' to `libxl__domain_suspend_state''
and move it to libxl_internal.h.
* Rename variables `si'' to `dss''.
* Change the stack-allocated state and callbacks from
struct suspendinfo si;
struct save_callbacks callbacks;
struct restore_callbacks callbacks;
to
libxl__domain_suspend_state dss[1];
struct save_callbacks callbacks[1];
struct restore_callbacks callbacks[1];
so that it may be referred to as a pointer variable everywhere.
* Rename the variable `flags'' (in libxl__domain_suspend_state) to
`xcflags'', to help distinguish it from the other `flags''
which is
passed in from the calling application in libxl_domain_suspend_info.
Abolish the local variable in libxl__domain_suspend_common, as it
can use the one in the dss.
* Move the prototypes of suspend-related functions in libxl_internal.h
to after the definition of the state struct.
* Replace several ctx variables with gc variables and
consequently references to ctx with CTX. Change references
to `dss->gc'' in the functional code to simply `gc''.
* Use LOG* rather than LIBXL__LOG* in a number of places.
* In libxl__domain_save_device_model use `rc'' instead of
`ret''.
* Introduce and use `gc'' and `domid'' in
libxl__domain_suspend_common_callback.
* Wrap some long lines.
* Add an extra pair of parens for clarity in a flag test.
* Remove two pointless casts from void* to a struct*.
No functional change whatsoever.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in v3:
* Abolish local variables `xcflags'' and `hvm'' in
libxl__domain_suspend_common; just use dss->xcflags and dss->hvm
instead and hence do not lose some of the changes to xcflags.
Changes in v2:
* Make callbacks into arrays (for pointerisation) too.
* Updated to cope with new remus code.
* Fixed typo in commit message.
---
tools/libxl/libxl_dom.c | 261 ++++++++++++++++++++----------------------
tools/libxl/libxl_internal.h | 30 ++++-
2 files changed, 151 insertions(+), 140 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 677db1d..e90030d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -470,7 +470,7 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data)
{
- libxl__gc *gc = (libxl__gc *) data;
+ libxl__gc *gc = data;
libxl_ctx *ctx = gc->owner;
int i, ret;
const uint8_t *ptr = buf;
@@ -531,7 +531,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
/* read signature */
int rc;
int hvm, pae, superpages;
- struct restore_callbacks callbacks;
+ struct restore_callbacks callbacks[1];
int no_incr_generationid;
switch (info->type) {
case LIBXL_DOMAIN_TYPE_HVM:
@@ -539,8 +539,8 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
superpages = 1;
pae = libxl_defbool_val(info->u.hvm.pae);
no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
- callbacks.toolstack_restore = libxl__toolstack_restore;
- callbacks.data = gc;
+ callbacks->toolstack_restore = libxl__toolstack_restore;
+ callbacks->data = gc;
break;
case LIBXL_DOMAIN_TYPE_PV:
hvm = 0;
@@ -556,7 +556,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
state->store_domid, state->console_port,
&state->console_mfn, state->console_domid,
hvm, pae, superpages, no_incr_generationid,
- &state->vm_generationid_addr,
&callbacks);
+ &state->vm_generationid_addr, callbacks);
if ( rc ) {
LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain");
return ERROR_FAIL;
@@ -564,33 +564,23 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
return 0;
}
-struct suspendinfo {
- libxl__gc *gc;
- xc_evtchn *xce; /* event channel handle */
- int suspend_eventchn;
- int domid;
- int hvm;
- unsigned int flags;
- int guest_responded;
- int save_fd; /* Migration stream fd (for Remus) */
- int interval; /* checkpoint interval (for Remus) */
-};
-
-static int libxl__domain_suspend_common_switch_qemu_logdirty(int domid,
unsigned int enable, void *data)
+static int libxl__domain_suspend_common_switch_qemu_logdirty
+ (int domid, unsigned int enable, void *data)
{
- struct suspendinfo *si = data;
- libxl_ctx *ctx = libxl__gc_owner(si->gc);
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
char *path;
bool rc;
- path = libxl__sprintf(si->gc,
"/local/domain/0/device-model/%u/logdirty/cmd", domid);
+ path = libxl__sprintf(gc,
+ "/local/domain/0/device-model/%u/logdirty/cmd",
domid);
if (!path)
return 1;
if (enable)
- rc = xs_write(ctx->xsh, XBT_NULL, path, "enable",
strlen("enable"));
+ rc = xs_write(CTX->xsh, XBT_NULL, path, "enable",
strlen("enable"));
else
- rc = xs_write(ctx->xsh, XBT_NULL, path, "disable",
strlen("disable"));
+ rc = xs_write(CTX->xsh, XBT_NULL, path, "disable",
strlen("disable"));
return rc ? 0 : 1;
}
@@ -645,53 +635,56 @@ int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid)
static int libxl__domain_suspend_common_callback(void *data)
{
- struct suspendinfo *si = data;
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
int ret;
char *state = "suspend";
int watchdog;
- libxl_ctx *ctx = libxl__gc_owner(si->gc);
xs_transaction_t t;
- if (si->hvm) {
- xc_get_hvm_param(ctx->xch, si->domid, HVM_PARAM_CALLBACK_IRQ,
&hvm_pvdrv);
- xc_get_hvm_param(ctx->xch, si->domid, HVM_PARAM_ACPI_S_STATE,
&hvm_s_state);
+ /* Convenience aliases */
+ const uint32_t domid = dss->domid;
+
+ if (dss->hvm) {
+ xc_get_hvm_param(CTX->xch, domid, HVM_PARAM_CALLBACK_IRQ,
&hvm_pvdrv);
+ xc_get_hvm_param(CTX->xch, domid, HVM_PARAM_ACPI_S_STATE,
&hvm_s_state);
}
- if ((hvm_s_state == 0) && (si->suspend_eventchn >= 0)) {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "issuing %s suspend request via
event channel",
- si->hvm ? "PVHVM" : "PV");
- ret = xc_evtchn_notify(si->xce, si->suspend_eventchn);
+ if ((hvm_s_state == 0) && (dss->suspend_eventchn >= 0)) {
+ LOG(DEBUG, "issuing %s suspend request via event channel",
+ dss->hvm ? "PVHVM" : "PV");
+ ret = xc_evtchn_notify(dss->xce, dss->suspend_eventchn);
if (ret < 0) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "xc_evtchn_notify failed
ret=%d", ret);
+ LOG(ERROR, "xc_evtchn_notify failed ret=%d", ret);
return 0;
}
- ret = xc_await_suspend(ctx->xch, si->xce,
si->suspend_eventchn);
+ ret = xc_await_suspend(CTX->xch, dss->xce,
dss->suspend_eventchn);
if (ret < 0) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "xc_await_suspend failed
ret=%d", ret);
+ LOG(ERROR, "xc_await_suspend failed ret=%d", ret);
return 0;
}
- si->guest_responded = 1;
+ dss->guest_responded = 1;
goto guest_suspended;
}
- if (si->hvm && (!hvm_pvdrv || hvm_s_state)) {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Calling xc_domain_shutdown on
HVM domain");
- xc_domain_shutdown(ctx->xch, si->domid, SHUTDOWN_suspend);
+ if (dss->hvm && (!hvm_pvdrv || hvm_s_state)) {
+ LOG(DEBUG, "Calling xc_domain_shutdown on HVM domain");
+ xc_domain_shutdown(CTX->xch, domid, SHUTDOWN_suspend);
/* The guest does not (need to) respond to this sort of request. */
- si->guest_responded = 1;
+ dss->guest_responded = 1;
} else {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "issuing %s suspend request via
XenBus control node",
- si->hvm ? "PVHVM" : "PV");
+ LOG(DEBUG, "issuing %s suspend request via XenBus control
node",
+ dss->hvm ? "PVHVM" : "PV");
- libxl__domain_pvcontrol_write(si->gc, XBT_NULL, si->domid,
"suspend");
+ libxl__domain_pvcontrol_write(gc, XBT_NULL, domid,
"suspend");
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "wait for the guest to
acknowledge suspend request");
+ LOG(DEBUG, "wait for the guest to acknowledge suspend
request");
watchdog = 60;
while (!strcmp(state, "suspend") && watchdog > 0)
{
usleep(100000);
- state = libxl__domain_pvcontrol_read(si->gc, XBT_NULL,
si->domid);
+ state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
if (!state) state = "";
watchdog--;
@@ -707,17 +700,17 @@ static int libxl__domain_suspend_common_callback(void
*data)
* at the last minute.
*/
if (!strcmp(state, "suspend")) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "guest didn''t
acknowledge suspend, cancelling request");
+ LOG(ERROR, "guest didn''t acknowledge suspend,
cancelling request");
retry_transaction:
- t = xs_transaction_start(ctx->xsh);
+ t = xs_transaction_start(CTX->xsh);
- state = libxl__domain_pvcontrol_read(si->gc, t, si->domid);
+ state = libxl__domain_pvcontrol_read(gc, t, domid);
if (!state) state = "";
if (!strcmp(state, "suspend"))
- libxl__domain_pvcontrol_write(si->gc, t, si->domid,
"");
+ libxl__domain_pvcontrol_write(gc, t, domid, "");
- if (!xs_transaction_end(ctx->xsh, t, 0))
+ if (!xs_transaction_end(CTX->xsh, t, 0))
if (errno == EAGAIN)
goto retry_transaction;
@@ -729,27 +722,29 @@ static int libxl__domain_suspend_common_callback(void
*data)
* case we lost the race while cancelling and should continue.
*/
if (!strcmp(state, "suspend")) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "guest didn''t
acknowledge suspend, request cancelled");
+ LOG(ERROR, "guest didn''t acknowledge suspend, request
cancelled");
return 0;
}
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "guest acknowledged suspend
request");
- si->guest_responded = 1;
+ LOG(DEBUG, "guest acknowledged suspend request");
+ dss->guest_responded = 1;
}
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "wait for the guest to
suspend");
+ LOG(DEBUG, "wait for the guest to suspend");
watchdog = 60;
while (watchdog > 0) {
xc_domaininfo_t info;
usleep(100000);
- ret = xc_domain_getinfolist(ctx->xch, si->domid, 1, &info);
- if (ret == 1 && info.domain == si->domid &&
info.flags & XEN_DOMINF_shutdown) {
+ ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);
+ if (ret == 1 && info.domain == domid &&
+ (info.flags & XEN_DOMINF_shutdown)) {
int shutdown_reason;
- shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
& XEN_DOMINF_shutdownmask;
+ shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
+ & XEN_DOMINF_shutdownmask;
if (shutdown_reason == SHUTDOWN_suspend) {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "guest has
suspended");
+ LOG(DEBUG, "guest has suspended");
goto guest_suspended;
}
}
@@ -757,15 +752,14 @@ static int libxl__domain_suspend_common_callback(void
*data)
watchdog--;
}
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "guest did not suspend");
+ LOG(ERROR, "guest did not suspend");
return 0;
guest_suspended:
- if (si->hvm) {
- ret = libxl__domain_suspend_device_model(si->gc, si->domid);
+ if (dss->hvm) {
+ ret = libxl__domain_suspend_device_model(dss->gc, dss->domid);
if (ret) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
- "libxl__domain_suspend_device_model failed
ret=%d", ret);
+ LOG(ERROR, "libxl__domain_suspend_device_model failed
ret=%d", ret);
return 0;
}
}
@@ -783,9 +777,8 @@ static inline char *save_helper(libxl__gc *gc, uint32_t
domid,
static int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
uint32_t *len, void *data)
{
- struct suspendinfo *si = (struct suspendinfo *) data;
- libxl__gc *gc = (libxl__gc *) si->gc;
- libxl_ctx *ctx = gc->owner;
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
int i = 0;
char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL;
unsigned int num = 0;
@@ -814,21 +807,21 @@ static int libxl__toolstack_save(uint32_t domid, uint8_t
**buf,
char *xs_path;
phys_offset = entries[i];
if (phys_offset == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "phys_offset %d is
NULL", i);
+ LOG(ERROR, "phys_offset %d is NULL", i);
return -1;
}
xs_path = save_helper(gc, domid, phys_offset, "start_addr");
start_addr = libxl__xs_read(gc, 0, xs_path);
if (start_addr == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "%s is NULL", xs_path);
+ LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
xs_path = save_helper(gc, domid, phys_offset, "size");
size = libxl__xs_read(gc, 0, xs_path);
if (size == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "%s is NULL", xs_path);
+ LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
@@ -864,11 +857,11 @@ static int libxl__remus_domain_suspend_callback(void
*data)
static int libxl__remus_domain_resume_callback(void *data)
{
- struct suspendinfo *si = data;
- libxl_ctx *ctx = libxl__gc_owner(si->gc);
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
/* Resumes the domain and the device model */
- if (libxl_domain_resume(ctx, si->domid, /* Fast Suspend */1))
+ if (libxl_domain_resume(CTX, dss->domid, /* Fast Suspend */1))
return 0;
/* TODO: Deal with disk. Start a new network output buffer */
@@ -877,15 +870,15 @@ static int libxl__remus_domain_resume_callback(void *data)
static int libxl__remus_domain_checkpoint_callback(void *data)
{
- struct suspendinfo *si = data;
+ libxl__domain_suspend_state *dss = data;
/* This would go into tailbuf. */
- if (si->hvm &&
- libxl__domain_save_device_model(si->gc, si->domid,
si->save_fd))
+ if (dss->hvm &&
+ libxl__domain_save_device_model(dss->gc, dss->domid,
dss->save_fd))
return 0;
/* TODO: Wait for disk and memory ack, release network buffer */
- usleep(si->interval * 1000);
+ usleep(dss->interval * 1000);
return 1;
}
@@ -894,12 +887,10 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
int live, int debug,
const libxl_domain_remus_info *r_info)
{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- int flags;
int port;
- struct save_callbacks callbacks;
- struct suspendinfo si;
- int hvm, rc = ERROR_FAIL;
+ struct save_callbacks callbacks[1];
+ libxl__domain_suspend_state dss[1];
+ int rc = ERROR_FAIL;
unsigned long vm_generationid_addr;
switch (type) {
@@ -912,82 +903,81 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
addr = libxl__xs_read(gc, XBT_NULL, path);
vm_generationid_addr = (addr) ? strtoul(addr, NULL, 0) : 0;
- hvm = 1;
+ dss->hvm = 1;
break;
}
case LIBXL_DOMAIN_TYPE_PV:
vm_generationid_addr = 0;
- hvm = 0;
+ dss->hvm = 0;
break;
default:
return ERROR_INVAL;
}
- memset(&si, 0, sizeof(si));
- flags = (live) ? XCFLAGS_LIVE : 0
+ dss->xcflags = (live) ? XCFLAGS_LIVE : 0
| (debug) ? XCFLAGS_DEBUG : 0
- | (hvm) ? XCFLAGS_HVM : 0;
+ | (dss->hvm) ? XCFLAGS_HVM : 0;
+
+ dss->domid = domid;
+ dss->gc = gc;
+ dss->suspend_eventchn = -1;
+ dss->guest_responded = 0;
if (r_info != NULL) {
- si.interval = r_info->interval;
+ dss->interval = r_info->interval;
if (r_info->compression)
- flags |= XCFLAGS_CHECKPOINT_COMPRESS;
- si.save_fd = fd;
+ dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
+ dss->save_fd = fd;
}
else
- si.save_fd = -1;
+ dss->save_fd = -1;
- si.domid = domid;
- si.flags = flags;
- si.hvm = hvm;
- si.gc = gc;
- si.suspend_eventchn = -1;
- si.guest_responded = 0;
-
- si.xce = xc_evtchn_open(NULL, 0);
- if (si.xce == NULL)
+ dss->xce = xc_evtchn_open(NULL, 0);
+ if (dss->xce == NULL)
goto out;
else
{
- port = xs_suspend_evtchn_port(si.domid);
+ port = xs_suspend_evtchn_port(dss->domid);
if (port >= 0) {
- si.suspend_eventchn = xc_suspend_evtchn_init(ctx->xch, si.xce,
si.domid, port);
+ dss->suspend_eventchn +
xc_suspend_evtchn_init(CTX->xch, dss->xce, dss->domid, port);
- if (si.suspend_eventchn < 0)
- LIBXL__LOG(ctx, LIBXL__LOG_WARNING, "Suspend event channel
initialization failed");
+ if (dss->suspend_eventchn < 0)
+ LOG(WARN, "Suspend event channel initialization
failed");
}
}
- memset(&callbacks, 0, sizeof(callbacks));
+ memset(callbacks, 0, sizeof(*callbacks));
if (r_info != NULL) {
- callbacks.suspend = libxl__remus_domain_suspend_callback;
- callbacks.postcopy = libxl__remus_domain_resume_callback;
- callbacks.checkpoint = libxl__remus_domain_checkpoint_callback;
+ callbacks->suspend = libxl__remus_domain_suspend_callback;
+ callbacks->postcopy = libxl__remus_domain_resume_callback;
+ callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
} else
- callbacks.suspend = libxl__domain_suspend_common_callback;
+ callbacks->suspend = libxl__domain_suspend_common_callback;
- callbacks.switch_qemu_logdirty =
libxl__domain_suspend_common_switch_qemu_logdirty;
- callbacks.toolstack_save = libxl__toolstack_save;
- callbacks.data = &si;
+ callbacks->switch_qemu_logdirty =
libxl__domain_suspend_common_switch_qemu_logdirty;
+ callbacks->toolstack_save = libxl__toolstack_save;
+ callbacks->data = dss;
- rc = xc_domain_save(ctx->xch, fd, domid, 0, 0, flags, &callbacks,
- hvm, vm_generationid_addr);
+ rc = xc_domain_save(CTX->xch, fd, domid, 0, 0, dss->xcflags,
callbacks,
+ dss->hvm, vm_generationid_addr);
if ( rc ) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "saving domain: %s",
- si.guest_responded ?
+ LOGE(ERROR, "saving domain: %s",
+ dss->guest_responded ?
"domain responded to suspend request" :
"domain did not respond to suspend
request");
- if ( !si.guest_responded )
+ if ( !dss->guest_responded )
rc = ERROR_GUEST_TIMEDOUT;
else
rc = ERROR_FAIL;
}
- if (si.suspend_eventchn > 0)
- xc_suspend_evtchn_release(ctx->xch, si.xce, domid,
si.suspend_eventchn);
- if (si.xce != NULL)
- xc_evtchn_close(si.xce);
+ if (dss->suspend_eventchn > 0)
+ xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
+ dss->suspend_eventchn);
+ if (dss->xce != NULL)
+ xc_evtchn_close(dss->xce);
out:
return rc;
@@ -995,8 +985,7 @@ out:
int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd)
{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- int ret, fd2 = -1, c;
+ int rc, fd2 = -1, c;
char buf[1024];
const char *filename = libxl__device_model_savefile(gc, domid);
struct stat st;
@@ -1004,46 +993,46 @@ int libxl__domain_save_device_model(libxl__gc *gc,
uint32_t domid, int fd)
if (stat(filename, &st) < 0)
{
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Unable to stat qemu save
file\n");
- ret = ERROR_FAIL;
+ LOG(ERROR, "Unable to stat qemu save file\n");
+ rc = ERROR_FAIL;
goto out;
}
qemu_state_len = st.st_size;
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Qemu state is %d bytes\n",
qemu_state_len);
+ LOG(DEBUG, "Qemu state is %d bytes\n", qemu_state_len);
- ret = libxl_write_exactly(ctx, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE),
+ rc = libxl_write_exactly(CTX, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE),
"saved-state file", "qemu
signature");
- if (ret)
+ if (rc)
goto out;
- ret = libxl_write_exactly(ctx, fd, &qemu_state_len,
sizeof(qemu_state_len),
+ rc = libxl_write_exactly(CTX, fd, &qemu_state_len,
sizeof(qemu_state_len),
"saved-state file", "saved-state
length");
- if (ret)
+ if (rc)
goto out;
fd2 = open(filename, O_RDONLY);
if (fd2 < 0) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Unable to open qemu save
file\n");
+ LOGE(ERROR, "Unable to open qemu save file\n");
goto out;
}
while ((c = read(fd2, buf, sizeof(buf))) != 0) {
if (c < 0) {
if (errno == EINTR)
continue;
- ret = errno;
+ rc = errno;
goto out;
}
- ret = libxl_write_exactly(
- ctx, fd, buf, c, "saved-state file", "qemu
state");
- if (ret)
+ rc = libxl_write_exactly(
+ CTX, fd, buf, c, "saved-state file", "qemu
state");
+ if (rc)
goto out;
}
- ret = 0;
+ rc = 0;
out:
if (fd2 >= 0) close(fd2);
unlink(filename);
- return ret;
+ return rc;
}
char *libxl__uuid2string(libxl__gc *gc, const libxl_uuid uuid)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index fa4c08f..f22bf94 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -786,10 +786,6 @@ _hidden int libxl__domain_restore_common(libxl__gc *gc,
uint32_t domid,
libxl_domain_build_info *info,
libxl__domain_build_state *state,
int fd);
-_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
- libxl_domain_type type,
- int live, int debug,
- const libxl_domain_remus_info
*r_info);
_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
@@ -1778,6 +1774,23 @@ _hidden void
libxl__datacopier_kill(libxl__datacopier_state *dc);
_hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
+/*----- Domain suspend (save) state structure -----*/
+
+typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
+
+struct libxl__domain_suspend_state {
+ libxl__gc *gc;
+ xc_evtchn *xce; /* event channel handle */
+ int suspend_eventchn;
+ int domid;
+ int hvm;
+ unsigned int xcflags;
+ int guest_responded;
+ int save_fd; /* Migration stream fd (for Remus) */
+ int interval; /* checkpoint interval (for Remus) */
+};
+
+
/*----- openpty -----*/
/*
@@ -1888,6 +1901,15 @@ struct libxl__domain_create_state {
* for the non-stubdom device model. */
};
+/*----- Domain suspend (save) functions -----*/
+
+_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
+ libxl_domain_type type,
+ int live, int debug,
+ const libxl_domain_remus_info
*r_info);
+
+
+
/*
* Convenience macros.
*/
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 04/21] libxl: domain restore: reshuffle, preparing for ao
We are going to arrange that libxl, instead of calling
xc_domain_restore, calls a stub function which forks and execs a
helper program, so that restore can be asynchronous rather than
blocking the whole toolstack.
This stub function will be called libxl__xc_domain_restore.
However, its prospective call site is unsuitable for a function which
needs to make a callback, and is buried in two nested single-call-site
functions which are logically part of the domain creation procedure.
So we first abolish those single-call-site functions, integrate their
contents into domain creation in their proper temporal order, and
break out libxl__xc_domain_restore ready for its reimplementation.
No functional change - just the following reorganisation:
* Abolish libxl__domain_restore_common, as it had only one caller.
Move its contents into (what was) domain_restore.
* There is a new stage function domcreate_rebuild_done containing what
used to be the bulk of domcreate_bootloader_done, since
domcreate_bootloader_done now simply starts the restore (or does the
rebuild) and arranges to call the next stage.
* Move the contents of domain_restore into its correct place in the
domain creation sequence. We put it inside
domcreate_bootloader_done, which now either calls
libxl__xc_domain_restore which will call the new function
domcreate_rebuild_done, or calls domcreate_rebuild_done directly.
* Various general-purpose local variables (`i'' etc.) and convenience
alias variables need to be shuffled about accordingly.
* Consequently libxl__toolstack_restore needs to gain external linkage
as it is now in a different file to its user.
* Move the xc_domain_save callbacks struct from the stack into
libxl__domain_create_state.
In general the moved code remains almost identical. Two returns in
what used to be libxl__domain_restore_common have been changed to set
the return value and "goto out", and the call sites for the abolished
and new functions have been adjusted.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in v2:
* Also move the save callbacks
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_create.c | 244 +++++++++++++++++++++++--------------
tools/libxl/libxl_dom.c | 45 +-------
tools/libxl/libxl_internal.h | 19 +++-
tools/libxl/libxl_save_callout.c | 37 ++++++
5 files changed, 206 insertions(+), 140 deletions(-)
create mode 100644 tools/libxl/libxl_save_callout.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index e7d5cc2..1d8b80a 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -67,6 +67,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o
libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o \
+ libxl_save_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 4456ae8..4439367 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -21,7 +21,6 @@
#include "libxl_arch.h"
#include <xc_dom.h>
-#include <xenguest.h>
void libxl_domain_config_init(libxl_domain_config *d_config)
{
@@ -317,89 +316,6 @@ out:
return ret;
}
-static int domain_restore(libxl__gc *gc, libxl_domain_build_info *info,
- uint32_t domid, int fd,
- libxl__domain_build_state *state)
-{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- char **vments = NULL, **localents = NULL;
- struct timeval start_time;
- int i, ret, esave, flags;
-
- ret = libxl__build_pre(gc, domid, info, state);
- if (ret)
- goto out;
-
- ret = libxl__domain_restore_common(gc, domid, info, state, fd);
- if (ret)
- goto out;
-
- gettimeofday(&start_time, NULL);
-
- switch (info->type) {
- case LIBXL_DOMAIN_TYPE_HVM:
- vments = libxl__calloc(gc, 7, sizeof(char *));
- vments[0] = "rtc/timeoffset";
- vments[1] = (info->u.hvm.timeoffset) ? info->u.hvm.timeoffset :
"";
- vments[2] = "image/ostype";
- vments[3] = "hvm";
- vments[4] = "start_time";
- vments[5] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
- break;
- case LIBXL_DOMAIN_TYPE_PV:
- vments = libxl__calloc(gc, 11, sizeof(char *));
- i = 0;
- vments[i++] = "image/ostype";
- vments[i++] = "linux";
- vments[i++] = "image/kernel";
- vments[i++] = (char *) state->pv_kernel.path;
- vments[i++] = "start_time";
- vments[i++] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
- if (state->pv_ramdisk.path) {
- vments[i++] = "image/ramdisk";
- vments[i++] = (char *) state->pv_ramdisk.path;
- }
- if (state->pv_cmdline) {
- vments[i++] = "image/cmdline";
- vments[i++] = (char *) state->pv_cmdline;
- }
- break;
- default:
- ret = ERROR_INVAL;
- goto out;
- }
- ret = libxl__build_post(gc, domid, info, state, vments, localents);
- if (ret)
- goto out;
-
- if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
- ret = asprintf(&state->saved_state,
- XC_DEVICE_MODEL_RESTORE_FILE".%d", domid);
- ret = (ret < 0) ? ERROR_FAIL : 0;
- }
-
-out:
- if (info->type == LIBXL_DOMAIN_TYPE_PV) {
- libxl__file_reference_unmap(&state->pv_kernel);
- libxl__file_reference_unmap(&state->pv_ramdisk);
- }
-
- esave = errno;
-
- flags = fcntl(fd, F_GETFL);
- if (flags == -1) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to get flags on
restore fd");
- } else {
- flags &= ~O_NONBLOCK;
- if (fcntl(fd, F_SETFL, flags) == -1)
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to put restore
fd"
- " back to blocking mode");
- }
-
- errno = esave;
- return ret;
-}
-
int libxl__domain_make(libxl__gc *gc, libxl_domain_create_info *info,
uint32_t *domid)
{
@@ -580,10 +496,13 @@ static void
domcreate_bootloader_console_available(libxl__egc *egc,
static void domcreate_bootloader_done(libxl__egc *egc,
libxl__bootloader_state *bl,
int rc);
-
static void domcreate_console_available(libxl__egc *egc,
libxl__domain_create_state *dcs);
+static void domcreate_rebuild_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int ret);
+
/* Our own function to clean up and call the user''s callback.
* The final call in the sequence. */
static void domcreate_complete(libxl__egc *egc,
@@ -671,20 +590,20 @@ static void domcreate_console_available(libxl__egc *egc,
static void domcreate_bootloader_done(libxl__egc *egc,
libxl__bootloader_state *bl,
- int ret)
+ int rc)
{
libxl__domain_create_state *dcs = CONTAINER_OF(bl, *dcs, bl);
STATE_AO_GC(bl->ao);
- int i;
/* convenience aliases */
const uint32_t domid = dcs->guest_domid;
libxl_domain_config *const d_config = dcs->guest_config;
+ libxl_domain_build_info *const info = &d_config->b_info;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- libxl_ctx *const ctx = CTX;
+ struct restore_callbacks *const callbacks = &dcs->callbacks;
- if (ret) goto error_out;
+ if (rc) domcreate_rebuild_done(egc, dcs, rc);
/* consume bootloader outputs. state->pv_{kernel,ramdisk} have
* been initialised by the bootloader already.
@@ -700,12 +619,153 @@ static void domcreate_bootloader_done(libxl__egc *egc,
dcs->dmss.dm.callback = domcreate_devmodel_started;
dcs->dmss.callback = domcreate_devmodel_started;
- if ( restore_fd >= 0 ) {
- ret = domain_restore(gc, &d_config->b_info, domid, restore_fd,
state);
+ if ( restore_fd < 0 ) {
+ rc = libxl__domain_build(gc, &d_config->b_info, domid, state);
+ domcreate_rebuild_done(egc, dcs, rc);
+ return;
+ }
+
+ /* Restore */
+
+ rc = libxl__build_pre(gc, domid, info, state);
+ if (rc)
+ goto out;
+
+ /* read signature */
+ int hvm, pae, superpages;
+ int no_incr_generationid;
+ switch (info->type) {
+ case LIBXL_DOMAIN_TYPE_HVM:
+ hvm = 1;
+ superpages = 1;
+ pae = libxl_defbool_val(info->u.hvm.pae);
+ no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
+ callbacks->toolstack_restore = libxl__toolstack_restore;
+ callbacks->data = gc;
+ break;
+ case LIBXL_DOMAIN_TYPE_PV:
+ hvm = 0;
+ superpages = 0;
+ pae = 1;
+ no_incr_generationid = 0;
+ break;
+ default:
+ rc = ERROR_INVAL;
+ goto out;
+ }
+ libxl__xc_domain_restore(egc, dcs,
+ hvm, pae, superpages, no_incr_generationid);
+ return;
+
+ out:
+ libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0);
+}
+
+void libxl__xc_domain_restore_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int ret, int retval, int errnoval)
+{
+ STATE_AO_GC(dcs->ao);
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ char **vments = NULL, **localents = NULL;
+ struct timeval start_time;
+ int i, esave, flags;
+
+ /* convenience aliases */
+ const uint32_t domid = dcs->guest_domid;
+ libxl_domain_config *const d_config = dcs->guest_config;
+ libxl_domain_build_info *const info = &d_config->b_info;
+ libxl__domain_build_state *const state = &dcs->build_state;
+ const int fd = dcs->restore_fd;
+
+ if (ret)
+ goto out;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "restoring domain");
+ ret = ERROR_FAIL;
+ goto out;
+ }
+
+ gettimeofday(&start_time, NULL);
+
+ switch (info->type) {
+ case LIBXL_DOMAIN_TYPE_HVM:
+ vments = libxl__calloc(gc, 7, sizeof(char *));
+ vments[0] = "rtc/timeoffset";
+ vments[1] = (info->u.hvm.timeoffset) ? info->u.hvm.timeoffset :
"";
+ vments[2] = "image/ostype";
+ vments[3] = "hvm";
+ vments[4] = "start_time";
+ vments[5] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
+ break;
+ case LIBXL_DOMAIN_TYPE_PV:
+ vments = libxl__calloc(gc, 11, sizeof(char *));
+ i = 0;
+ vments[i++] = "image/ostype";
+ vments[i++] = "linux";
+ vments[i++] = "image/kernel";
+ vments[i++] = (char *) state->pv_kernel.path;
+ vments[i++] = "start_time";
+ vments[i++] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
+ if (state->pv_ramdisk.path) {
+ vments[i++] = "image/ramdisk";
+ vments[i++] = (char *) state->pv_ramdisk.path;
+ }
+ if (state->pv_cmdline) {
+ vments[i++] = "image/cmdline";
+ vments[i++] = (char *) state->pv_cmdline;
+ }
+ break;
+ default:
+ ret = ERROR_INVAL;
+ goto out;
+ }
+ ret = libxl__build_post(gc, domid, info, state, vments, localents);
+ if (ret)
+ goto out;
+
+ if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
+ ret = asprintf(&state->saved_state,
+ XC_DEVICE_MODEL_RESTORE_FILE".%d", domid);
+ ret = (ret < 0) ? ERROR_FAIL : 0;
+ }
+
+out:
+ if (info->type == LIBXL_DOMAIN_TYPE_PV) {
+ libxl__file_reference_unmap(&state->pv_kernel);
+ libxl__file_reference_unmap(&state->pv_ramdisk);
+ }
+
+ esave = errno;
+
+ flags = fcntl(fd, F_GETFL);
+ if (flags == -1) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to get flags on
restore fd");
} else {
- ret = libxl__domain_build(gc, &d_config->b_info, domid, state);
+ flags &= ~O_NONBLOCK;
+ if (fcntl(fd, F_SETFL, flags) == -1)
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to put restore
fd"
+ " back to blocking mode");
}
+ errno = esave;
+ domcreate_rebuild_done(egc, dcs, ret);
+}
+
+static void domcreate_rebuild_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int ret)
+{
+ STATE_AO_GC(dcs->ao);
+ int i;
+
+ /* convenience aliases */
+ const uint32_t domid = dcs->guest_domid;
+ libxl_domain_config *const d_config = dcs->guest_config;
+ libxl__domain_build_state *const state = &dcs->build_state;
+ libxl_ctx *const ctx = CTX;
+
if (ret) {
LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "cannot (re-)build domain:
%d", ret);
ret = ERROR_FAIL;
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index e90030d..1d4e809 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -19,7 +19,6 @@
#include <xenctrl.h>
#include <xc_dom.h>
-#include <xenguest.h>
#include <xen/hvm/hvm_info_table.h>
@@ -467,7 +466,7 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
domid, phys_offset, node);
}
-static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
+int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data)
{
libxl__gc *gc = data;
@@ -522,48 +521,6 @@ static int libxl__toolstack_restore(uint32_t domid, const
uint8_t *buf,
return 0;
}
-int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid,
- libxl_domain_build_info *info,
- libxl__domain_build_state *state,
- int fd)
-{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- /* read signature */
- int rc;
- int hvm, pae, superpages;
- struct restore_callbacks callbacks[1];
- int no_incr_generationid;
- switch (info->type) {
- case LIBXL_DOMAIN_TYPE_HVM:
- hvm = 1;
- superpages = 1;
- pae = libxl_defbool_val(info->u.hvm.pae);
- no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
- callbacks->toolstack_restore = libxl__toolstack_restore;
- callbacks->data = gc;
- break;
- case LIBXL_DOMAIN_TYPE_PV:
- hvm = 0;
- superpages = 0;
- pae = 1;
- no_incr_generationid = 0;
- break;
- default:
- return ERROR_INVAL;
- }
- rc = xc_domain_restore(ctx->xch, fd, domid,
- state->store_port, &state->store_mfn,
- state->store_domid, state->console_port,
- &state->console_mfn, state->console_domid,
- hvm, pae, superpages, no_incr_generationid,
- &state->vm_generationid_addr, callbacks);
- if ( rc ) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain");
- return ERROR_FAIL;
- }
- return 0;
-}
-
static int libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data)
{
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index f22bf94..28478ea 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -46,6 +46,7 @@
#include <xenstore.h>
#include <xenctrl.h>
+#include <xenguest.h>
#include "xentoollog.h"
@@ -782,10 +783,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t
domid,
const char *old_name, const char *new_name,
xs_transaction_t trans);
-_hidden int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid,
- libxl_domain_build_info *info,
- libxl__domain_build_state *state,
- int fd);
+_hidden int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
+ uint32_t size, void *data);
_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
@@ -1899,6 +1898,7 @@ struct libxl__domain_create_state {
libxl__stub_dm_spawn_state dmss;
/* If we''re not doing stubdom, we use only dmss.dm,
* for the non-stubdom device model. */
+ struct restore_callbacks callbacks;
};
/*----- Domain suspend (save) functions -----*/
@@ -1908,6 +1908,17 @@ _hidden int libxl__domain_suspend_common(libxl__gc *gc,
uint32_t domid, int fd,
int live, int debug,
const libxl_domain_remus_info
*r_info);
+/* calls libxl__xc_domain_restore_done when done */
+_hidden void libxl__xc_domain_restore(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int hvm, int pae, int superpages,
+ int no_incr_generationid);
+/* If rc==0 then retval is the return value from xc_domain_save
+ * and errnoval is the errno value it provided.
+ * If rc!=0, retval and errnoval are undefined. */
+_hidden void libxl__xc_domain_restore_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int rc, int retval, int errnoval);
/*
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
new file mode 100644
index 0000000..2f8db9f
--- /dev/null
+++ b/tools/libxl/libxl_save_callout.c
@@ -0,0 +1,37 @@
+/*
+ * Copyright (C) 2012 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h"
+
+#include "libxl_internal.h"
+
+void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
+ int hvm, int pae, int superpages,
+ int no_incr_generationid)
+{
+ STATE_AO_GC(dcs->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = dcs->guest_domid;
+ const int restore_fd = dcs->restore_fd;
+ libxl__domain_build_state *const state = &dcs->build_state;
+
+ int r = xc_domain_restore(CTX->xch, restore_fd, domid,
+ state->store_port, &state->store_mfn,
+ state->store_domid, state->console_port,
+ &state->console_mfn,
state->console_domid,
+ hvm, pae, superpages, no_incr_generationid,
+ &state->vm_generationid_addr,
&dcs->callbacks);
+ libxl__xc_domain_restore_done(egc, dcs, 0, r, errno);
+}
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 05/21] libxl: domain save: API changes for asynchrony
Change the internal and external APIs for domain save (suspend) to be
capable of asynchronous operation. The implementation remains
synchronous. The interfaces surrounding device model saving are still
synchronous.
Public API changes:
* libxl_domain_save takes an ao_how.
* libxl_domain_remus_start takes an ao_how. If the
libxl_domain_remus_info is NULL, we abort rather than returning an
error.
* The `suspend_callback'' function passed to libxl_domain_save is
never called by the existing implementation in libxl. Abolish it.
* libxl_domain_save takes its flags parameter as an argument.
Thus libxl_domain_suspend_info is abolished.
* XL_SUSPEND_* flags renamed to LIBXL_SAVE_*.
* Callers in xl updated.
Internal code restructuring:
* libxl__domain_suspend_state member types and names rationalised.
* libxl__domain_suspend renamed from libxl__domain_suspend_common.
(_common here actually meant "internal function").
* libxl__domain_suspend takes a libxl__domain_suspend_state, which
where the parameters to the operation are filled in by the caller.
* xc_domain_save is now called via libxl__xc_domain_save which can
itself become asynchronous.
* Consequently, libxl__domain_suspend is split into two functions at
the callback boundary; the second half is
libxl__xc_domain_save_done.
* libxl__domain_save_device_model is now called by the actual
implementation rather than by the public wrapper. It is already in
its proper place in the domain save execution sequence. So
officially make it part of that execution sequence, renaming it to
domain_save_device_model.
* Effectively, rewrite the public wrapper functions
libxl_domain_suspend and libxl_domain_remus_start.
* Remove a needless #include <xenctrl.h>
* libxl__domain_suspend aborts on unexpected domain types rather
than mysteriously returning EINVAL.
* struct save_callbacks moved from the stack to the dss.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v4:
* libxl_domain_suspend now handles error from libxl__domain_type
in the correct way. (Hunk brought forward from domain type
fixup patch.)
* Comment clarifies that libxl__domain_suspend calls dss->callback
when done.
Changes in v3:
* Remove `hvm'' and `xcflags'' args to libxl__xc_domain_save.
Instead,
just use the values from the dss.
Changes in v2:
* Move save_callbacks too.
* Merge with Remus changes.
* Improvements to commit message.
* Do not rename libxl_domain_suspend any more.
---
tools/libxl/libxl.c | 94 +++++++++++++++++++++---------
tools/libxl/libxl.h | 22 ++++----
tools/libxl/libxl_dom.c | 121 +++++++++++++++++++++++++++-----------
tools/libxl/libxl_internal.h | 45 ++++++++++++---
tools/libxl/libxl_save_callout.c | 11 ++++
tools/libxl/xl_cmdimpl.c | 9 +--
6 files changed, 214 insertions(+), 88 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 6215923..f7fcf1b 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -648,32 +648,51 @@ libxl_vminfo * libxl_list_vm(libxl_ctx *ctx, int *nb_vm)
return ptr;
}
+static void remus_crashed_cb(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc);
+
/* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
- uint32_t domid, int send_fd, int recv_fd)
+ uint32_t domid, int send_fd, int recv_fd,
+ const libxl_asyncop_how *ao_how)
{
- GC_INIT(ctx);
- libxl_domain_type type = libxl__domain_type(gc, domid);
- int rc = 0;
+ AO_CREATE(ctx, domid, ao_how);
+ libxl__domain_suspend_state *dss;
+ int rc;
+ libxl_domain_type type = libxl__domain_type(gc, domid);
if (type == LIBXL_DOMAIN_TYPE_INVALID) {
rc = ERROR_FAIL;
- goto remus_fail;
+ goto out;
}
- if (info == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
- "No remus_info structure supplied for domain %d",
domid);
- rc = ERROR_INVAL;
- goto remus_fail;
- }
+ GCNEW(dss);
+ dss->ao = ao;
+ dss->callback = remus_crashed_cb;
+ dss->domid = domid;
+ dss->fd = send_fd;
+ /* TODO do something with recv_fd */
+ dss->type = type;
+ dss->live = 1;
+ dss->debug = 0;
+ dss->remus = info;
+
+ assert(info);
/* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */
/* Point of no return */
- rc = libxl__domain_suspend_common(gc, domid, send_fd, type, /* live */ 1,
- /* debug */ 0, info);
+ libxl__domain_suspend(egc, dss);
+ return AO_INPROGRESS;
+
+ out:
+ return AO_ABORT(rc);
+}
+static void remus_crashed_cb(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
+{
+ STATE_AO_GC(dss->ao);
/*
* With Remus, if we reach this point, it means either
* backup died or some network error occurred preventing us
@@ -683,27 +702,46 @@ int libxl_domain_remus_start(libxl_ctx *ctx,
libxl_domain_remus_info *info,
/* TBD: Remus cleanup - i.e. detach qdisc, release other
* resources.
*/
- remus_fail:
- GC_FREE;
- return rc;
+ libxl__ao_complete(egc, ao, rc);
}
-int libxl_domain_suspend(libxl_ctx *ctx, libxl_domain_suspend_info *info,
- uint32_t domid, int fd)
+static void domain_suspend_cb(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
{
- GC_INIT(ctx);
+ STATE_AO_GC(dss->ao);
+ libxl__ao_complete(egc,ao,rc);
+
+}
+
+int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags,
+ const libxl_asyncop_how *ao_how)
+{
+ AO_CREATE(ctx, domid, ao_how);
+ int rc;
+
libxl_domain_type type = libxl__domain_type(gc, domid);
- int live = info != NULL && info->flags & XL_SUSPEND_LIVE;
- int debug = info != NULL && info->flags & XL_SUSPEND_DEBUG;
- int rc = 0;
+ if (type == LIBXL_DOMAIN_TYPE_INVALID) {
+ rc = ERROR_FAIL;
+ goto out_err;
+ }
- rc = libxl__domain_suspend_common(gc, domid, fd, type, live, debug,
- /* No Remus */ NULL);
+ libxl__domain_suspend_state *dss;
+ GCNEW(dss);
- if (!rc && type == LIBXL_DOMAIN_TYPE_HVM)
- rc = libxl__domain_save_device_model(gc, domid, fd);
- GC_FREE;
- return rc;
+ dss->ao = ao;
+ dss->callback = domain_suspend_cb;
+
+ dss->domid = domid;
+ dss->fd = fd;
+ dss->type = type;
+ dss->live = flags & LIBXL_SUSPEND_LIVE;
+ dss->debug = flags & LIBXL_SUSPEND_DEBUG;
+
+ libxl__domain_suspend(egc, dss);
+ return AO_INPROGRESS;
+
+ out_err:
+ return AO_ABORT(rc);
}
int libxl_domain_pause(libxl_ctx *ctx, uint32_t domid)
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 05f0e01..10d7115 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -347,13 +347,6 @@ typedef struct libxl__ctx libxl_ctx;
const libxl_version_info* libxl_get_version_info(libxl_ctx *ctx);
-typedef struct {
-#define XL_SUSPEND_DEBUG 1
-#define XL_SUSPEND_LIVE 2
- int flags;
- int (*suspend_callback)(void *, int);
-} libxl_domain_suspend_info;
-
enum {
ERROR_NONSPECIFIC = -1,
ERROR_VERSION = -2,
@@ -514,16 +507,23 @@ int libxl_domain_create_restore(libxl_ctx *ctx,
libxl_domain_config *d_config,
void libxl_domain_config_init(libxl_domain_config *d_config);
void libxl_domain_config_dispose(libxl_domain_config *d_config);
-int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
- uint32_t domid, int send_fd, int recv_fd);
-int libxl_domain_suspend(libxl_ctx *ctx, libxl_domain_suspend_info *info,
- uint32_t domid, int fd);
+
+int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd,
+ int flags, /* LIBXL_SUSPEND_* */
+ const libxl_asyncop_how *ao_how);
+#define LIBXL_SUSPEND_DEBUG 1
+#define LIBXL_SUSPEND_LIVE 2
/* @param suspend_cancel [from xenctrl.h:xc_domain_resume( @param fast )]
* If this parameter is true, use co-operative resume. The guest
* must support this.
*/
int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int suspend_cancel);
+
+int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
+ uint32_t domid, int send_fd, int recv_fd,
+ const libxl_asyncop_how *ao_how);
+
int libxl_domain_shutdown(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_reboot(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_destroy(libxl_ctx *ctx, uint32_t domid);
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 1d4e809..b48452c 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -17,13 +17,11 @@
#include <glob.h>
-#include <xenctrl.h>
-#include <xc_dom.h>
+#include "libxl_internal.h"
+#include <xc_dom.h>
#include <xen/hvm/hvm_info_table.h>
-#include "libxl_internal.h"
-
libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid)
{
libxl_ctx *ctx = libxl__gc_owner(gc);
@@ -521,11 +519,18 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t
*buf,
return 0;
}
-static int libxl__domain_suspend_common_switch_qemu_logdirty
+/*==================== Domain suspend (save) ====================*/
+
+static void domain_suspend_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc);
+
+/*----- callbacks, called by xc_domain_save -----*/
+
+int libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
char *path;
bool rc;
@@ -590,10 +595,10 @@ int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid)
return 0;
}
-static int libxl__domain_suspend_common_callback(void *data)
+int libxl__domain_suspend_common_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
int ret;
char *state = "suspend";
@@ -714,7 +719,7 @@ static int libxl__domain_suspend_common_callback(void *data)
guest_suspended:
if (dss->hvm) {
- ret = libxl__domain_suspend_device_model(dss->gc, dss->domid);
+ ret = libxl__domain_suspend_device_model(gc, dss->domid);
if (ret) {
LOG(ERROR, "libxl__domain_suspend_device_model failed
ret=%d", ret);
return 0;
@@ -731,11 +736,11 @@ static inline char *save_helper(libxl__gc *gc, uint32_t
domid,
domid, phys_offset, node);
}
-static int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
+int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
uint32_t *len, void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
int i = 0;
char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL;
unsigned int num = 0;
@@ -806,6 +811,8 @@ static int libxl__toolstack_save(uint32_t domid, uint8_t
**buf,
return 0;
}
+/*----- remus callbacks -----*/
+
static int libxl__remus_domain_suspend_callback(void *data)
{
/* TODO: Issue disk and network checkpoint reqs. */
@@ -815,7 +822,7 @@ static int libxl__remus_domain_suspend_callback(void *data)
static int libxl__remus_domain_resume_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
/* Resumes the domain and the device model */
if (libxl_domain_resume(CTX, dss->domid, /* Fast Suspend */1))
@@ -828,10 +835,11 @@ static int libxl__remus_domain_resume_callback(void *data)
static int libxl__remus_domain_checkpoint_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
+ STATE_AO_GC(dss->ao);
/* This would go into tailbuf. */
if (dss->hvm &&
- libxl__domain_save_device_model(dss->gc, dss->domid,
dss->save_fd))
+ libxl__domain_save_device_model(gc, dss->domid, dss->fd))
return 0;
/* TODO: Wait for disk and memory ack, release network buffer */
@@ -839,17 +847,23 @@ static int libxl__remus_domain_checkpoint_callback(void
*data)
return 1;
}
-int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
- libxl_domain_type type,
- int live, int debug,
- const libxl_domain_remus_info *r_info)
+/*----- main code for suspending, in order of execution -----*/
+
+void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
{
+ STATE_AO_GC(dss->ao);
int port;
- struct save_callbacks callbacks[1];
- libxl__domain_suspend_state dss[1];
int rc = ERROR_FAIL;
unsigned long vm_generationid_addr;
+ /* Convenience aliases */
+ const uint32_t domid = dss->domid;
+ const libxl_domain_type type = dss->type;
+ const int live = dss->live;
+ const int debug = dss->debug;
+ const libxl_domain_remus_info *const r_info = dss->remus;
+ struct save_callbacks *const callbacks = &dss->callbacks;
+
switch (type) {
case LIBXL_DOMAIN_TYPE_HVM: {
char *path;
@@ -868,15 +882,13 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
dss->hvm = 0;
break;
default:
- return ERROR_INVAL;
+ abort();
}
dss->xcflags = (live) ? XCFLAGS_LIVE : 0
| (debug) ? XCFLAGS_DEBUG : 0
| (dss->hvm) ? XCFLAGS_HVM : 0;
- dss->domid = domid;
- dss->gc = gc;
dss->suspend_eventchn = -1;
dss->guest_responded = 0;
@@ -884,10 +896,7 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
dss->interval = r_info->interval;
if (r_info->compression)
dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
- dss->save_fd = fd;
}
- else
- dss->save_fd = -1;
dss->xce = xc_evtchn_open(NULL, 0);
if (dss->xce == NULL)
@@ -917,10 +926,28 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
callbacks->toolstack_save = libxl__toolstack_save;
callbacks->data = dss;
- rc = xc_domain_save(CTX->xch, fd, domid, 0, 0, dss->xcflags,
callbacks,
- dss->hvm, vm_generationid_addr);
- if ( rc ) {
- LOGE(ERROR, "saving domain: %s",
+ libxl__xc_domain_save(egc, dss, vm_generationid_addr);
+ return;
+
+ out:
+ domain_suspend_done(egc, dss, rc);
+}
+
+void libxl__xc_domain_save_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ int rc, int retval, int errnoval)
+{
+ STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases */
+ const libxl_domain_type type = dss->type;
+ const uint32_t domid = dss->domid;
+
+ if (rc)
+ goto out;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "saving domain: %s",
dss->guest_responded ?
"domain responded to suspend request" :
"domain did not respond to suspend
request");
@@ -928,16 +955,21 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
rc = ERROR_GUEST_TIMEDOUT;
else
rc = ERROR_FAIL;
+ goto out;
}
- if (dss->suspend_eventchn > 0)
- xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
- dss->suspend_eventchn);
- if (dss->xce != NULL)
- xc_evtchn_close(dss->xce);
+ if (type == LIBXL_DOMAIN_TYPE_HVM) {
+ rc = libxl__domain_suspend_device_model(gc, domid);
+ if (rc) goto out;
+
+ rc = libxl__domain_save_device_model(gc, domid, dss->fd);
+ if (rc) goto out;
+ }
+
+ rc = 0;
out:
- return rc;
+ domain_suspend_done(egc, dss, rc);
}
int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd)
@@ -992,6 +1024,25 @@ out:
return rc;
}
+static void domain_suspend_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
+{
+ STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = dss->domid;
+
+ if (dss->suspend_eventchn > 0)
+ xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
+ dss->suspend_eventchn);
+ if (dss->xce != NULL)
+ xc_evtchn_close(dss->xce);
+
+ dss->callback(egc, dss, rc);
+}
+
+/*==================== Miscellaneous ====================*/
+
char *libxl__uuid2string(libxl__gc *gc, const libxl_uuid uuid)
{
char *s = libxl__sprintf(gc, LIBXL_UUID_FMT, LIBXL_UUID_BYTES(uuid));
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 28478ea..7cf1b04 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1777,16 +1777,28 @@ _hidden int
libxl__datacopier_start(libxl__datacopier_state *dc);
typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
+typedef void libxl__domain_suspend_cb(libxl__egc*,
+ libxl__domain_suspend_state*, int rc);
+
struct libxl__domain_suspend_state {
- libxl__gc *gc;
+ /* set by caller of libxl__domain_suspend */
+ libxl__ao *ao;
+ libxl__domain_suspend_cb *callback;
+
+ uint32_t domid;
+ int fd;
+ libxl_domain_type type;
+ int live;
+ int debug;
+ const libxl_domain_remus_info *remus;
+ /* private */
xc_evtchn *xce; /* event channel handle */
int suspend_eventchn;
- int domid;
int hvm;
- unsigned int xcflags;
+ int xcflags;
int guest_responded;
- int save_fd; /* Migration stream fd (for Remus) */
int interval; /* checkpoint interval (for Remus) */
+ struct save_callbacks callbacks;
};
@@ -1903,10 +1915,27 @@ struct libxl__domain_create_state {
/*----- Domain suspend (save) functions -----*/
-_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
- libxl_domain_type type,
- int live, int debug,
- const libxl_domain_remus_info
*r_info);
+/* calls dss->callback when done */
+_hidden void libxl__domain_suspend(libxl__egc *egc,
+ libxl__domain_suspend_state *dss);
+
+
+/* calls libxl__xc_domain_suspend_done when done */
+_hidden void libxl__xc_domain_save(libxl__egc*, libxl__domain_suspend_state*,
+ unsigned long vm_generationid_addr);
+/* If rc==0 then retval is the return value from xc_domain_save
+ * and errnoval is the errno value it provided.
+ * If rc!=0, retval and errnoval are undefined. */
+_hidden void libxl__xc_domain_save_done(libxl__egc*,
+ libxl__domain_suspend_state*,
+ int rc, int retval, int errnoval);
+
+_hidden int libxl__domain_suspend_common_callback(void *data);
+_hidden int libxl__domain_suspend_common_switch_qemu_logdirty
+ (int domid, unsigned int enable, void *data);
+_hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
+ uint32_t *len, void *data);
+
/* calls libxl__xc_domain_restore_done when done */
_hidden void libxl__xc_domain_restore(libxl__egc *egc,
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 2f8db9f..1b481ab 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -35,3 +35,14 @@ void libxl__xc_domain_restore(libxl__egc *egc,
libxl__domain_create_state *dcs,
&state->vm_generationid_addr,
&dcs->callbacks);
libxl__xc_domain_restore_done(egc, dcs, 0, r, errno);
}
+
+void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
+ unsigned long vm_generationid_addr)
+{
+ STATE_AO_GC(dss->ao);
+ int r;
+
+ r = xc_domain_save(CTX->xch, dss->fd, dss->domid, 0, 0,
dss->xcflags,
+ &dss->callbacks, dss->hvm,
vm_generationid_addr);
+ libxl__xc_domain_save_done(egc, dss, 0, r, errno);
+}
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 3ea95ef..19daa1c 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -2846,7 +2846,7 @@ static int save_domain(const char *p, const char
*filename, int checkpoint,
save_domain_core_writeconfig(fd, filename, config_data, config_len);
- CHK_ERRNO(libxl_domain_suspend(ctx, NULL, domid, fd));
+ CHK_ERRNO(libxl_domain_suspend(ctx, domid, fd, 0, NULL));
close(fd);
if (checkpoint)
@@ -3008,7 +3008,6 @@ static void migrate_domain(const char *domain_spec, const
char *rune,
pid_t child = -1;
int rc;
int send_fd = -1, recv_fd = -1;
- libxl_domain_suspend_info suspinfo;
char *away_domname;
char rc_buf;
uint8_t *config_data;
@@ -3030,9 +3029,7 @@ static void migrate_domain(const char *domain_spec, const
char *rune,
xtl_stdiostream_adjust_flags(logger, XTL_STDIOSTREAM_HIDE_PROGRESS, 0);
- memset(&suspinfo, 0, sizeof(suspinfo));
- suspinfo.flags |= XL_SUSPEND_LIVE;
- rc = libxl_domain_suspend(ctx, &suspinfo, domid, send_fd);
+ rc = libxl_domain_suspend(ctx, domid, send_fd, LIBXL_SUSPEND_LIVE, NULL);
if (rc) {
fprintf(stderr, "migration sender: libxl_domain_suspend
failed"
" (rc=%d)\n", rc);
@@ -6604,7 +6601,7 @@ int main_remus(int argc, char **argv)
}
/* Point of no return */
- rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd);
+ rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd,
0);
/* If we are here, it means backup has failed/domain suspend failed.
* Try to resume the domain and exit gracefully.
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 06/21] libxl: domain save/restore: run in a separate process
libxenctrl expects to be able to simply run the save or restore
operation synchronously. This won''t work well in a process which is
trying to handle multiple domains.
The options are:
- Block such a whole process (eg, the whole of libvirt) while
migration completes (or until it fails).
- Create a thread to run xc_domain_save and xc_domain_restore on.
This is quite unpalatable. Multithreaded programming is error
prone enough without generating threads in libraries, particularly
if the thread does some very complex operation.
- Fork and run the operation in the child without execing. This is
no good because we would need to negotiate with the caller about
fds we would inherit (and we might be a very large process).
- Fork and exec a helper.
Of these options the latter is the most palatable.
Consequently:
* A new helper program libxl-save-helper (which does both save and
restore). It will be installed in /usr/lib/xen/bin. It does not
link against libxl, only libxc, and its error handling does not
need to be very advanced. It does contain a plumbing through of
the logging interface into the callback stream.
* A small ad-hoc protocol between the helper and libxl which allows
log messages and the libxc callbacks to be passed up and down.
Protocol doc comment is in libxl_save_helper.c.
* To avoid a lot of tedium the marshalling boilerplate (stubs for the
helper and the callback decoder for libxl) is generated with a
small perl script.
* Implement new functionality to spawn the helper, monitor its
output, provide responses, and check on its exit status.
* The functions libxl__xc_domain_restore_done and
libxl__xc_domain_save_done now turn out to want be called in the
same place. So make their state argument a void* so that the two
functions are type compatible.
The domain save path still writes the qemu savefile synchronously.
This will need to be fixed in a subsequent patch.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v4:
* Migration stream fd is handled specially by the run_helper
function, rather than simply being a numarg. Specifically:
- dup it to a safe fd number if necessary.
- clear cloexec flag fd before execing helper
* Toolstack data fd argument to run_helper replaced with
generic preserve_fds array, which get cloexec cleared.
* libxl__xc_domain_save uses supplied callback function pointer,
rather than calling libxl__toolstack_save directly;
toolstack data save callback is only supplied to libxc if
in-libxl caller supplied a callback.
* libxl-save-helper is not needlessly linked against libxl.
* Code which prepares pipes for helper clarified.
* Deal properly with, and log properly, POLLPRI/POLLERR on
pipe to save helper.
* Spelling fix in perl script comment.
* In message generator, use better names for the ends of serial
conditional here documents.
* Makefile does $(INSTALL_DIR) $(DESTDIR)$(PRIVATE_BINDIR)
Changes in v3:
* Suppress errno value in debug message when helper reports successful
completion.
* Significant consequential changes to cope with changes to
earlier patches in the series.
Changes in v2:
* Helper path can be overridden by an environment variable for testing.
* Add a couple of debug logging messages re toolstack data.
* Fixes from testing.
* Helper protocol message lengths (and numbers) are 16-bit which
more clearly avoids piling lots of junk on the stack.
* Merged with remus changes.
* Callback implementations in libxl now called via pointers
so remus can have its own callbacks.
* Better namespace prefixes on autogenerated names etc.
* Autogenerator can generate debugging printfs too.
---
.gitignore | 1 +
.hgignore | 2 +
tools/libxl/Makefile | 22 ++-
tools/libxl/libxl_create.c | 22 ++-
tools/libxl/libxl_dom.c | 36 ++--
tools/libxl/libxl_internal.h | 56 +++++-
tools/libxl/libxl_save_callout.c | 366 ++++++++++++++++++++++++++++++++-
tools/libxl/libxl_save_helper.c | 281 +++++++++++++++++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 397 ++++++++++++++++++++++++++++++++++++
9 files changed, 1145 insertions(+), 38 deletions(-)
create mode 100644 tools/libxl/libxl_save_helper.c
create mode 100755 tools/libxl/libxl_save_msgs_gen.pl
diff --git a/.gitignore b/.gitignore
index 7770e54..3451e52 100644
--- a/.gitignore
+++ b/.gitignore
@@ -353,6 +353,7 @@ tools/libxl/_*.[ch]
tools/libxl/testidl
tools/libxl/testidl.c
tools/libxl/*.pyc
+tools/libxl/libxl-save-helper
tools/blktap2/control/tap-ctl
tools/firmware/etherboot/eb-roms.h
tools/firmware/etherboot/gpxe-git-snapshot.tar.gz
diff --git a/.hgignore b/.hgignore
index 27d8f79..05304ea 100644
--- a/.hgignore
+++ b/.hgignore
@@ -180,9 +180,11 @@
^tools/libxl/_.*\.c$
^tools/libxl/libxlu_cfg_y\.output$
^tools/libxl/xl$
+^tools/libxl/libxl-save-helper$
^tools/libxl/testidl$
^tools/libxl/testidl\.c$
^tools/libxl/tmp\..*$
+^tools/libxl/.*\.new$
^tools/libvchan/vchan-node[12]$
^tools/libaio/src/.*\.ol$
^tools/libaio/src/.*\.os$
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 1d8b80a..ddc2624 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -67,25 +67,30 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o
libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o \
- libxl_save_callout.o \
+ libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
$(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
$(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include $(XEN_ROOT)/tools/config.h
-AUTOINCS= libxlu_cfg_y.h libxlu_cfg_l.h _libxl_list.h _paths.h
+AUTOINCS= libxlu_cfg_y.h libxlu_cfg_l.h _libxl_list.h _paths.h \
+ _libxl_save_msgs_callout.h _libxl_save_msgs_helper.h
AUTOSRCS= libxlu_cfg_y.c libxlu_cfg_l.c
+AUTOSRCS += _libxl_save_msgs_callout.c _libxl_save_msgs_helper.c
LIBXLU_OBJS = libxlu_cfg_y.o libxlu_cfg_l.o libxlu_cfg.o \
libxlu_disk_l.o libxlu_disk.o libxlu_vif.o libxlu_pci.o
$(LIBXLU_OBJS): CFLAGS += $(CFLAGS_libxenctrl) # For xentoollog.h
-CLIENTS = xl testidl
+CLIENTS = xl testidl libxl-save-helper
XL_OBJS = xl.o xl_cmdimpl.o xl_cmdtable.o xl_sxp.o
$(XL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) # For xentoollog.h
$(XL_OBJS): CFLAGS += $(CFLAGS_libxenlight)
$(XL_OBJS): CFLAGS += -include $(XEN_ROOT)/tools/config.h # libxl_json.h needs
it.
+SAVE_HELPER_OBJS = libxl_save_helper.o _libxl_save_msgs_helper.o
+$(SAVE_HELPER_OBJS): CFLAGS += $(CFLAGS_libxenctrl)
+
testidl.o: CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenlight)
testidl.c: libxl_types.idl gentest.py libxl.h $(AUTOINCS)
$(PYTHON) gentest.py libxl_types.idl testidl.c.new
@@ -117,6 +122,12 @@ _libxl_list.h:
$(XEN_INCLUDE)/xen-external/bsd-sys-queue-h-seddery $(XEN_INCLUDE
perl $^ --prefix=libxl >$@.new
$(call move-if-changed,$@.new,$@)
+_libxl_save_msgs_helper.c _libxl_save_msgs_callout.c \
+_libxl_save_msgs_helper.h _libxl_save_msgs_callout.h: \
+ libxl_save_msgs_gen.pl
+ $(PERL) -w $< $@ >$@.new
+ $(call move-if-changed,$@.new,$@)
+
libxl.h: _libxl_types.h
libxl_json.h: _libxl_types_json.h
libxl_internal.h: _libxl_types_internal.h _paths.h
@@ -159,6 +170,9 @@ libxlutil.a: $(LIBXLU_OBJS)
xl: $(XL_OBJS) libxlutil.so libxenlight.so
$(CC) $(LDFLAGS) -o $@ $(XL_OBJS) libxlutil.so $(LDLIBS_libxenlight)
$(LDLIBS_libxenctrl) -lyajl $(APPEND_LDFLAGS)
+libxl-save-helper: $(SAVE_HELPER_OBJS) libxenlight.so
+ $(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS) $(LDLIBS_libxenctrl)
$(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
+
testidl: testidl.o libxlutil.so libxenlight.so
$(CC) $(LDFLAGS) -o $@ testidl.o libxlutil.so $(LDLIBS_libxenlight)
$(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
@@ -169,7 +183,9 @@ install: all
$(INSTALL_DIR) $(DESTDIR)$(INCLUDEDIR)
$(INSTALL_DIR) $(DESTDIR)$(BASH_COMPLETION_DIR)
$(INSTALL_DIR) $(DESTDIR)$(XEN_RUN_DIR)
+ $(INSTALL_DIR) $(DESTDIR)$(PRIVATE_BINDIR)
$(INSTALL_PROG) xl $(DESTDIR)$(SBINDIR)
+ $(INSTALL_PROG) libxl-save-helper $(DESTDIR)$(PRIVATE_BINDIR)
$(INSTALL_PROG) libxenlight.so.$(MAJOR).$(MINOR) $(DESTDIR)$(LIBDIR)
ln -sf libxenlight.so.$(MAJOR).$(MINOR)
$(DESTDIR)$(LIBDIR)/libxenlight.so.$(MAJOR)
ln -sf libxenlight.so.$(MAJOR) $(DESTDIR)$(LIBDIR)/libxenlight.so
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 4439367..00705d8 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -601,7 +601,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
libxl_domain_build_info *const info = &d_config->b_info;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- struct restore_callbacks *const callbacks = &dcs->callbacks;
+ libxl__srm_restore_autogen_callbacks *const callbacks +
&dcs->shs.callbacks.restore.a;
if (rc) domcreate_rebuild_done(egc, dcs, rc);
@@ -641,7 +642,6 @@ static void domcreate_bootloader_done(libxl__egc *egc,
pae = libxl_defbool_val(info->u.hvm.pae);
no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
callbacks->toolstack_restore = libxl__toolstack_restore;
- callbacks->data = gc;
break;
case LIBXL_DOMAIN_TYPE_PV:
hvm = 0;
@@ -661,10 +661,24 @@ static void domcreate_bootloader_done(libxl__egc *egc,
libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0);
}
-void libxl__xc_domain_restore_done(libxl__egc *egc,
- libxl__domain_create_state *dcs,
+void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
+ unsigned long console_mfn, unsigned long genidad, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
+ STATE_AO_GC(dcs->ao);
+ libxl__domain_build_state *const state = &dcs->build_state;
+
+ state->store_mfn = store_mfn;
+ state->console_mfn = console_mfn;
+ state->vm_generationid_addr = genidad;
+ shs->need_results = 0;
+}
+
+void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int ret, int retval, int errnoval)
{
+ libxl__domain_create_state *dcs = dcs_void;
STATE_AO_GC(dcs->ao);
libxl_ctx *ctx = libxl__gc_owner(gc);
char **vments = NULL, **localents = NULL;
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index b48452c..fea0c94 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -465,16 +465,20 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
}
int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
- uint32_t size, void *data)
+ uint32_t size, void *user)
{
- libxl__gc *gc = data;
- libxl_ctx *ctx = gc->owner;
+ libxl__save_helper_state *shs = user;
+ libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
+ STATE_AO_GC(dcs->ao);
+ libxl_ctx *ctx = CTX;
int i, ret;
const uint8_t *ptr = buf;
uint32_t count = 0, version = 0;
struct libxl__physmap_info* pi;
char *xs_path;
+ LOG(DEBUG,"domain=%"PRIu32" toolstack data
size=%"PRIu32, domid, size);
+
if (size < sizeof(version) + sizeof(count)) {
LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "wrong size");
return -1;
@@ -527,9 +531,10 @@ static void domain_suspend_done(libxl__egc *egc,
/*----- callbacks, called by xc_domain_save -----*/
int libxl__domain_suspend_common_switch_qemu_logdirty
- (int domid, unsigned int enable, void *data)
+ (int domid, unsigned enable, void *user)
{
- libxl__domain_suspend_state *dss = data;
+ libxl__save_helper_state *shs = user;
+ libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
STATE_AO_GC(dss->ao);
char *path;
bool rc;
@@ -595,9 +600,10 @@ int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid)
return 0;
}
-int libxl__domain_suspend_common_callback(void *data)
+int libxl__domain_suspend_common_callback(void *user)
{
- libxl__domain_suspend_state *dss = data;
+ libxl__save_helper_state *shs = user;
+ libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
STATE_AO_GC(dss->ao);
unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
int ret;
@@ -737,9 +743,9 @@ static inline char *save_helper(libxl__gc *gc, uint32_t
domid,
}
int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
- uint32_t *len, void *data)
+ uint32_t *len, void *dss_void)
{
- libxl__domain_suspend_state *dss = data;
+ libxl__domain_suspend_state *dss = dss_void;
STATE_AO_GC(dss->ao);
int i = 0;
char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL;
@@ -808,6 +814,8 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
ptr += sizeof(struct libxl__physmap_info) + namelen;
}
+ LOG(DEBUG,"domain=%"PRIu32" toolstack data
size=%"PRIu32, domid, *len);
+
return 0;
}
@@ -862,7 +870,8 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
const int live = dss->live;
const int debug = dss->debug;
const libxl_domain_remus_info *const r_info = dss->remus;
- struct save_callbacks *const callbacks = &dss->callbacks;
+ libxl__srm_save_autogen_callbacks *const callbacks +
&dss->shs.callbacks.save.a;
switch (type) {
case LIBXL_DOMAIN_TYPE_HVM: {
@@ -923,8 +932,7 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
callbacks->suspend = libxl__domain_suspend_common_callback;
callbacks->switch_qemu_logdirty =
libxl__domain_suspend_common_switch_qemu_logdirty;
- callbacks->toolstack_save = libxl__toolstack_save;
- callbacks->data = dss;
+ dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
libxl__xc_domain_save(egc, dss, vm_generationid_addr);
return;
@@ -933,10 +941,10 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
domain_suspend_done(egc, dss, rc);
}
-void libxl__xc_domain_save_done(libxl__egc *egc,
- libxl__domain_suspend_state *dss,
+void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
int rc, int retval, int errnoval)
{
+ libxl__domain_suspend_state *dss = dss_void;
STATE_AO_GC(dss->ao);
/* Convenience aliases */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 7cf1b04..1a7b526 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -54,6 +54,7 @@
#include "libxl.h"
#include "_paths.h"
+#include "_libxl_save_msgs_callout.h"
#if __GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)
#define _hidden __attribute__((visibility("hidden")))
@@ -1773,6 +1774,51 @@ _hidden void
libxl__datacopier_kill(libxl__datacopier_state *dc);
_hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
+/*----- Save/restore helper (used by creation and suspend) -----*/
+
+typedef struct libxl__srm_save_callbacks {
+ libxl__srm_save_autogen_callbacks a;
+ int (*toolstack_save)(uint32_t domid, uint8_t **buf,
+ uint32_t *len, void *data);
+} libxl__srm_save_callbacks;
+
+typedef struct libxl__srm_restore_callbacks {
+ libxl__srm_restore_autogen_callbacks a;
+} libxl__srm_restore_callbacks;
+
+/* a pointer to this struct is also passed as "user" to the
+ * save callout helper callback functions */
+typedef struct libxl__save_helper_state {
+ /* public, caller of run_helper initialises */
+ libxl__ao *ao;
+ uint32_t domid;
+ union {
+ libxl__srm_save_callbacks save;
+ libxl__srm_restore_callbacks restore;
+ } callbacks;
+ int (*recv_callback)(const unsigned char *msg, uint32_t len, void *user);
+ void (*completion_callback)(libxl__egc *egc, void *caller_state,
+ int rc, int retval, int errnoval);
+ void *caller_state;
+ int need_results; /* set to 0 or 1 by caller of run_helper;
+ * if set to 1 then the ultimate caller''s
+ * results function must set it to 0 */
+ /* private */
+ int rc;
+ int completed; /* retval/errnoval valid iff completed */
+ int retval, errnoval; /* from xc_domain_save / xc_domain_restore */
+ libxl__carefd *pipes[2]; /* 0 = helper''s stdin, 1 =
helper''s stdout */
+ libxl__ev_fd readable;
+ libxl__ev_child child;
+ const char *stdin_what, *stdout_what;
+ FILE *toolstack_data_file;
+
+ libxl__egc *egc; /* valid only for duration of each event callback;
+ * is here in this struct for the benefit of the
+ * marshalling and xc callback functions */
+} libxl__save_helper_state;
+
+
/*----- Domain suspend (save) state structure -----*/
typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
@@ -1798,7 +1844,7 @@ struct libxl__domain_suspend_state {
int xcflags;
int guest_responded;
int interval; /* checkpoint interval (for Remus) */
- struct save_callbacks callbacks;
+ libxl__save_helper_state shs;
};
@@ -1910,7 +1956,7 @@ struct libxl__domain_create_state {
libxl__stub_dm_spawn_state dmss;
/* If we''re not doing stubdom, we use only dmss.dm,
* for the non-stubdom device model. */
- struct restore_callbacks callbacks;
+ libxl__save_helper_state shs;
};
/*----- Domain suspend (save) functions -----*/
@@ -1926,8 +1972,7 @@ _hidden void libxl__xc_domain_save(libxl__egc*,
libxl__domain_suspend_state*,
/* If rc==0 then retval is the return value from xc_domain_save
* and errnoval is the errno value it provided.
* If rc!=0, retval and errnoval are undefined. */
-_hidden void libxl__xc_domain_save_done(libxl__egc*,
- libxl__domain_suspend_state*,
+_hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
int rc, int retval, int errnoval);
_hidden int libxl__domain_suspend_common_callback(void *data);
@@ -1945,8 +1990,7 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc,
/* If rc==0 then retval is the return value from xc_domain_save
* and errnoval is the errno value it provided.
* If rc!=0, retval and errnoval are undefined. */
-_hidden void libxl__xc_domain_restore_done(libxl__egc *egc,
- libxl__domain_create_state *dcs,
+_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int rc, int retval, int errnoval);
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 1b481ab..007d61c 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -16,6 +16,30 @@
#include "libxl_internal.h"
+/* stream_fd is as from the caller (eventually, the application).
+ * It may be 0, 1 or 2, in which case we need to dup it elsewhere.
+ * The actual fd value is not included in the supplied argnums; rather
+ * it will be automatically supplied by run_helper as the 2nd argument.
+ *
+ * preserve_fds are fds that the caller is intending to pass to the
+ * helper so which need cloexec clearing. They may not be 0, 1 or 2.
+ * An entry may be -1 in which case it will be ignored.
+ */
+static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
+ const char *mode_arg,
+ int stream_fd,
+ const int *preserve_fds, int num_preserve_fds,
+ const unsigned long *argnums, int num_argnums);
+
+static void helper_failed(libxl__egc*, libxl__save_helper_state *shs, int rc);
+static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
+ int fd, short events, short revents);
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status);
+static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs);
+
+/*----- entrypoints -----*/
+
void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
int hvm, int pae, int superpages,
int no_incr_generationid)
@@ -27,22 +51,342 @@ void libxl__xc_domain_restore(libxl__egc *egc,
libxl__domain_create_state *dcs,
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- int r = xc_domain_restore(CTX->xch, restore_fd, domid,
- state->store_port, &state->store_mfn,
- state->store_domid, state->console_port,
- &state->console_mfn,
state->console_domid,
- hvm, pae, superpages, no_incr_generationid,
- &state->vm_generationid_addr,
&dcs->callbacks);
- libxl__xc_domain_restore_done(egc, dcs, 0, r, errno);
+ unsigned cbflags = libxl__srm_callout_enumcallbacks_restore
+ (&dcs->shs.callbacks.restore.a);
+
+ const unsigned long argnums[] = {
+ domid,
+ state->store_port,
+ state->store_domid, state->console_port,
+ state->console_domid,
+ hvm, pae, superpages, no_incr_generationid,
+ cbflags,
+ };
+
+ dcs->shs.ao = ao;
+ dcs->shs.domid = domid;
+ dcs->shs.recv_callback = libxl__srm_callout_received_restore;
+ dcs->shs.completion_callback = libxl__xc_domain_restore_done;
+ dcs->shs.caller_state = dcs;
+ dcs->shs.need_results = 1;
+ dcs->shs.toolstack_data_file = 0;
+
+ run_helper(egc, &dcs->shs, "--restore-domain", restore_fd,
0,0,
+ argnums, ARRAY_SIZE(argnums));
}
void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
unsigned long vm_generationid_addr)
{
STATE_AO_GC(dss->ao);
- int r;
+ int r, rc, toolstack_data_fd = -1;
+ uint32_t toolstack_data_len = 0;
+
+ /* Resources we need to free */
+ uint8_t *toolstack_data_buf = 0;
+
+ unsigned cbflags = libxl__srm_callout_enumcallbacks_save
+ (&dss->shs.callbacks.save.a);
+
+ if (dss->shs.callbacks.save.toolstack_save) {
+ r = dss->shs.callbacks.save.toolstack_save
+ (dss->domid, &toolstack_data_buf, &toolstack_data_len,
dss);
+ if (r) { rc = ERROR_FAIL; goto out; }
+
+ dss->shs.toolstack_data_file = tmpfile();
+ if (!dss->shs.toolstack_data_file) {
+ LOGE(ERROR, "cannot create toolstack data tmpfile");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ toolstack_data_fd = fileno(dss->shs.toolstack_data_file);
+
+ r = libxl_write_exactly(CTX, toolstack_data_fd,
+ toolstack_data_buf, toolstack_data_len,
+ "toolstack data tmpfile", 0);
+ if (r) { rc = ERROR_FAIL; goto out; }
+
+ r = lseek(toolstack_data_fd, 0, SEEK_SET);
+ if (r) {
+ LOGE(ERROR, "rewind toolstack data tmpfile");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ }
+
+ const unsigned long argnums[] = {
+ dss->domid, 0, 0, dss->xcflags, dss->hvm,
vm_generationid_addr,
+ toolstack_data_fd, toolstack_data_len,
+ cbflags,
+ };
+
+ dss->shs.ao = ao;
+ dss->shs.domid = dss->domid;
+ dss->shs.recv_callback = libxl__srm_callout_received_save;
+ dss->shs.completion_callback = libxl__xc_domain_save_done;
+ dss->shs.caller_state = dss;
+ dss->shs.need_results = 0;
+
+ free(toolstack_data_buf);
+
+ run_helper(egc, &dss->shs, "--save-domain", dss->fd,
+ &toolstack_data_fd, 1,
+ argnums, ARRAY_SIZE(argnums));
+ return;
+
+ out:
+ free(toolstack_data_buf);
+ if (dss->shs.toolstack_data_file)
fclose(dss->shs.toolstack_data_file);
+
+ libxl__xc_domain_save_done(egc, dss, rc, 0, 0);
+}
+
+
+/*----- helper execution -----*/
+
+static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
+ const char *mode_arg, int stream_fd,
+ const int *preserve_fds, int num_preserve_fds,
+ const unsigned long *argnums, int num_argnums)
+{
+ STATE_AO_GC(shs->ao);
+ const char *args[4 + num_argnums];
+ const char **arg = args;
+ int i, rc;
+
+ /* Resources we must free */
+ libxl__carefd *childs_pipes[2] = { 0,0 };
+
+ /* Convenience aliases */
+ const uint32_t domid = shs->domid;
+
+ shs->rc = 0;
+ shs->completed = 0;
+ shs->pipes[0] = shs->pipes[1] = 0;
+ libxl__ev_fd_init(&shs->readable);
+ libxl__ev_child_init(&shs->child);
+
+ shs->stdin_what = GCSPRINTF("domain %"PRIu32"
save/restore helper"
+ " stdin pipe", domid);
+ shs->stdout_what = GCSPRINTF("domain %"PRIu32"
save/restore helper"
+ " stdout pipe", domid);
+
+ *arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC "/"
"libxl-save-helper";
+ *arg++ = mode_arg;
+ const char **stream_fd_arg = arg++;
+ for (i=0; i<num_argnums; i++)
+ *arg++ = GCSPRINTF("%lu", argnums[i]);
+ *arg++ = 0;
+ assert(arg == args + ARRAY_SIZE(args));
+
+ libxl__carefd_begin();
+ int childfd;
+ for (childfd=0; childfd<2; childfd++) {
+ /* Setting up the pipe for the child''s fd childfd */
+ int fds[2];
+ if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; }
+ int childs_end = childfd==0 ? 0 /*read*/ : 1 /*write*/;
+ int our_end = childfd==0 ? 1 /*write*/ : 0 /*read*/;
+ childs_pipes[childfd] = libxl__carefd_record(CTX, fds[childs_end]);
+ shs->pipes[childfd] = libxl__carefd_record(CTX, fds[our_end]);
+ }
+ libxl__carefd_unlock();
+
+ pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited);
+ if (!pid) {
+ if (stream_fd <= 2) {
+ stream_fd = dup(stream_fd);
+ if (stream_fd < 0) {
+ LOGE(ERROR,"dup migration stream fd");
+ exit(-1);
+ }
+ }
+ libxl_fd_set_cloexec(CTX, stream_fd, 0);
+ *stream_fd_arg = GCSPRINTF("%d", stream_fd);
+
+ for (i=0; i<num_preserve_fds; i++)
+ if (preserve_fds[i] >= 0)
+ libxl_fd_set_cloexec(CTX, preserve_fds[i], 0);
+
+ libxl__exec(gc,
+ libxl__carefd_fd(childs_pipes[0]),
+ libxl__carefd_fd(childs_pipes[1]),
+ -1,
+ args[0], (char**)args, 0);
+ }
+
+ libxl__carefd_close(childs_pipes[0]);
+ libxl__carefd_close(childs_pipes[1]);
+
+ rc = libxl__ev_fd_register(gc, &shs->readable,
helper_stdout_readable,
+ libxl__carefd_fd(shs->pipes[1]),
POLLIN|POLLPRI);
+ if (rc) goto out;
+ return;
+
+ out:
+ libxl__carefd_close(childs_pipes[0]);
+ libxl__carefd_close(childs_pipes[1]);
+ helper_failed(egc, shs, rc);;
+}
+
+static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
+ int rc)
+{
+ STATE_AO_GC(shs->ao);
+
+ if (!shs->rc)
+ shs->rc = rc;
+
+ libxl__ev_fd_deregister(gc, &shs->readable);
+
+ if (!libxl__ev_child_inuse(&shs->child)) {
+ helper_done(egc, shs);
+ return;
+ }
+
+ int r = kill(shs->child.pid, SIGKILL);
+ if (r) LOGE(WARN, "failed to kill save/restore helper [%lu]",
+ (unsigned long)shs->child.pid);
+}
+
+static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
+ int fd, short events, short revents)
+{
+ libxl__save_helper_state *shs = CONTAINER_OF(ev, *shs, readable);
+ STATE_AO_GC(shs->ao);
+ int rc, errnoval;
+
+ if (revents & (POLLERR|POLLPRI)) {
+ LOG(ERROR, "%s signaled POLLERR|POLLPRI (%#x)",
+ shs->stdout_what, revents);
+ rc = ERROR_FAIL;
+ out:
+ /* this is here because otherwise we bypass the decl of msg[] */
+ helper_failed(egc, shs, rc);
+ return;
+ }
+
+ uint16_t msglen;
+ errnoval = libxl_read_exactly(CTX, fd, &msglen, sizeof(msglen),
+ shs->stdout_what, "ipc msg
header");
+ if (errnoval) { rc = ERROR_FAIL; goto out; }
+
+ unsigned char msg[msglen];
+ errnoval = libxl_read_exactly(CTX, fd, msg, msglen,
+ shs->stdout_what, "ipc msg
body");
+ if (errnoval) { rc = ERROR_FAIL; goto out; }
+
+ shs->egc = egc;
+ shs->recv_callback(msg, msglen, shs);
+ shs->egc = 0;
+ return;
+}
+
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status)
+{
+ libxl__save_helper_state *shs = CONTAINER_OF(ch, *shs, child);
+ STATE_AO_GC(shs->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = shs->domid;
+
+ const char *what + GCSPRINTF("domain %"PRIu32"
save/restore helper", domid);
+
+ if (status) {
+ libxl_report_child_exitstatus(CTX, XTL_ERROR, what, pid, status);
+ shs->rc = ERROR_FAIL;
+ }
+
+ if (shs->need_results) {
+ if (!shs->rc)
+ LOG(ERROR,"%s exited without providing results",what);
+ shs->rc = ERROR_FAIL;
+ }
+
+ if (!shs->completed) {
+ if (!shs->rc)
+ LOG(ERROR,"%s exited without signaling completion",what);
+ shs->rc = ERROR_FAIL;
+ }
+
+ helper_done(egc, shs);
+ return;
+}
+
+static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs)
+{
+ STATE_AO_GC(shs->ao);
+
+ libxl__ev_fd_deregister(gc, &shs->readable);
+ libxl__carefd_close(shs->pipes[0]); shs->pipes[0] = 0;
+ libxl__carefd_close(shs->pipes[1]); shs->pipes[1] = 0;
+ assert(!libxl__ev_child_inuse(&shs->child));
+ if (shs->toolstack_data_file) fclose(shs->toolstack_data_file);
+
+ shs->egc = egc;
+ shs->completion_callback(egc, shs->caller_state,
+ shs->rc, shs->retval, shs->errnoval);
+ shs->egc = 0;
+}
+
+/*----- generic helpers for the autogenerated code -----*/
+
+const libxl__srm_save_autogen_callbacks*
+libxl__srm_callout_get_callbacks_save(void *user)
+{
+ libxl__save_helper_state *shs = user;
+ return &shs->callbacks.save.a;
+}
+
+const libxl__srm_restore_autogen_callbacks*
+libxl__srm_callout_get_callbacks_restore(void *user)
+{
+ libxl__save_helper_state *shs = user;
+ return &shs->callbacks.restore.a;
+}
+
+void libxl__srm_callout_sendreply(int r, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ libxl__egc *egc = shs->egc;
+ STATE_AO_GC(shs->ao);
+ int errnoval;
+
+ errnoval = libxl_write_exactly(CTX, libxl__carefd_fd(shs->pipes[0]),
+ &r, sizeof(r), shs->stdin_what,
+ "callback return value");
+ if (errnoval)
+ helper_failed(egc, shs, ERROR_FAIL);
+}
+
+void libxl__srm_callout_callback_log(uint32_t level, uint32_t errnoval,
+ const char *context, const char *formatted, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ STATE_AO_GC(shs->ao);
+ xtl_log(CTX->lg, level, errnoval, context, "%s", formatted);
+}
+
+void libxl__srm_callout_callback_progress(const char *context,
+ const char *doing_what, unsigned long done,
+ unsigned long total, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ STATE_AO_GC(shs->ao);
+ xtl_progress(CTX->lg, context, doing_what, done, total);
+}
+
+int libxl__srm_callout_callback_complete(int retval, int errnoval,
+ void *user)
+{
+ libxl__save_helper_state *shs = user;
+ STATE_AO_GC(shs->ao);
- r = xc_domain_save(CTX->xch, dss->fd, dss->domid, 0, 0,
dss->xcflags,
- &dss->callbacks, dss->hvm,
vm_generationid_addr);
- libxl__xc_domain_save_done(egc, dss, 0, r, errno);
+ shs->completed = 1;
+ shs->retval = retval;
+ shs->errnoval = errnoval;
+ libxl__ev_fd_deregister(gc, &shs->readable);
+ return 0;
}
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
new file mode 100644
index 0000000..3bdfa28
--- /dev/null
+++ b/tools/libxl/libxl_save_helper.c
@@ -0,0 +1,281 @@
+/*
+ * Copyright (C) 2012 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+/*
+ * The libxl-save-helper utility speaks a protocol to its caller for
+ * the callbacks. The protocol is as follows.
+ *
+ * The helper talks on stdin and stdout, in binary in machine
+ * endianness. The helper speaks first, and only when it has a
+ * callback to make. It writes a 16-bit number being the message
+ * length, and then the message body.
+ *
+ * Each message starts with a 16-bit number indicating which of the
+ * messages it is, and then some arguments in a binary marshalled form.
+ * If the callback does not need a reply (it returns void), the helper
+ * just continues. Otherwise the helper waits for its caller to send a
+ * single int which is to be the return value from the callback.
+ *
+ * Where feasible the stubs and callbacks have prototypes identical to
+ * those required by xc_domain_save and xc_domain_restore, so that the
+ * autogenerated functions can be used/provided directly.
+ *
+ * The actual messages are in the array @msgs in libxl_save_msgs_gen.pl
+ */
+
+#include "libxl_osdeps.h"
+
+#include <stdlib.h>
+#include <unistd.h>
+#include <assert.h>
+#include <inttypes.h>
+
+#include "libxl.h"
+
+#include "xenctrl.h"
+#include "xenguest.h"
+#include "_libxl_save_msgs_helper.h"
+
+/*----- globals -----*/
+
+static const char *program = "libxl-save-helper";
+static xentoollog_logger *logger;
+static xc_interface *xch;
+
+/*----- error handling -----*/
+
+static void fail(int errnoval, const char *fmt, ...)
+ __attribute__((noreturn,format(printf,2,3)));
+static void fail(int errnoval, const char *fmt, ...)
+{
+ va_list al;
+ va_start(al,fmt);
+ xtl_logv(logger,XTL_ERROR,errnoval,program,fmt,al);
+ exit(-1);
+}
+
+static int read_exactly(int fd, void *buf, size_t len)
+/* returns 0 if we get eof, even if we got it midway through; 1 if ok */
+{
+ while (len) {
+ ssize_t r = read(fd, buf, len);
+ if (r<=0) return r;
+ assert(r <= len);
+ len -= r;
+ buf = (char*)buf + r;
+ }
+ return 1;
+}
+
+static void *xmalloc(size_t sz)
+{
+ if (!sz) return 0;
+ void *r = malloc(sz);
+ if (!r) { perror("memory allocation failed"); exit(-1); }
+ return r;
+}
+
+/*----- logger -----*/
+
+typedef struct {
+ xentoollog_logger vtable;
+} xentoollog_logger_tellparent;
+
+static void tellparent_vmessage(xentoollog_logger *logger_in,
+ xentoollog_level level,
+ int errnoval,
+ const char *context,
+ const char *format,
+ va_list al)
+{
+ char *formatted;
+ int r = vasprintf(&formatted, format, al);
+ if (r < 0) { perror("memory allocation failed during
logging"); exit(-1); }
+ helper_stub_log(level, errnoval, context, formatted, 0);
+ free(formatted);
+}
+
+static void tellparent_progress(struct xentoollog_logger *logger_in,
+ const char *context,
+ const char *doing_what, int percent,
+ unsigned long done, unsigned long total)
+{
+ helper_stub_progress(context, doing_what, done, total, 0);
+}
+
+static void tellparent_destroy(struct xentoollog_logger *logger_in)
+{
+ abort();
+}
+
+static xentoollog_logger_tellparent *createlogger_tellparent(void)
+{
+ xentoollog_logger_tellparent newlogger;
+ return XTL_NEW_LOGGER(tellparent, newlogger);
+}
+
+/*----- helper functions called by autogenerated stubs -----*/
+
+unsigned char * helper_allocbuf(int len, void *user)
+{
+ return xmalloc(len);
+}
+
+static void transmit(const unsigned char *msg, int len, void *user)
+{
+ while (len) {
+ int r = write(1, msg, len);
+ if (r<0) { perror("write"); exit(-1); }
+ assert(r >= 0);
+ assert(r <= len);
+ len -= r;
+ msg += r;
+ }
+}
+
+void helper_transmitmsg(unsigned char *msg_freed, int len_in, void *user)
+{
+ assert(len_in < 64*1024);
+ uint16_t len = len_in;
+ transmit((const void*)&len, sizeof(len), user);
+ transmit(msg_freed, len, user);
+ free(msg_freed);
+}
+
+int helper_getreply(void *user)
+{
+ int v;
+ int r = read_exactly(0, &v, sizeof(v));
+ if (r<=0) exit(-2);
+ return v;
+}
+
+/*----- other callbacks -----*/
+
+static int toolstack_save_fd;
+static uint32_t toolstack_save_len;
+
+static int toolstack_save_cb(uint32_t domid, uint8_t **buf,
+ uint32_t *len, void *data)
+{
+ assert(toolstack_save_fd > 0);
+
+ *buf = xmalloc(toolstack_save_len);
+ int r = read_exactly(toolstack_save_fd, *buf, toolstack_save_len);
+ if (r<0) fail(errno,"read toolstack data");
+ if (r==0) fail(0,"read toolstack data eof");
+
+ toolstack_save_fd = -1;
+ *len = toolstack_save_len;
+ return 0;
+}
+
+static void startup(const char *op) {
+ logger = (xentoollog_logger*)createlogger_tellparent();
+ if (!logger) {
+ fprintf(stderr, "%s: cannot initialise logger\n", program);
+ exit(-1);
+ }
+
+ xtl_log(logger,XTL_DEBUG,0,program,"starting %s",op);
+
+ xch = xc_interface_open(logger,logger,0);
+ if (!xch) fail(errno,"xc_interface_open failed");
+}
+
+static void complete(int retval) {
+ int errnoval = retval ? errno : 0; /* suppress irrelevant errnos */
+ xtl_log(logger,XTL_DEBUG,errnoval,program,"complete
r=%d",retval);
+ helper_stub_complete(retval,errnoval,0);
+ exit(0);
+}
+
+static struct save_callbacks helper_save_callbacks;
+static struct restore_callbacks helper_restore_callbacks;
+
+int main(int argc, char **argv)
+{
+ int r;
+
+#define NEXTARG (++argv, assert(*argv), *argv)
+
+ const char *mode = *++argv;
+ assert(mode);
+
+ if (!strcmp(mode,"--save-domain")) {
+
+ int io_fd = atoi(NEXTARG);
+ uint32_t dom = strtoul(NEXTARG,0,10);
+ uint32_t max_iters = strtoul(NEXTARG,0,10);
+ uint32_t max_factor = strtoul(NEXTARG,0,10);
+ uint32_t flags = strtoul(NEXTARG,0,10);
+ int hvm = atoi(NEXTARG);
+ unsigned long genidad = strtoul(NEXTARG,0,10);
+ toolstack_save_fd = atoi(NEXTARG);
+ toolstack_save_len = strtoul(NEXTARG,0,10);
+ unsigned cbflags = strtoul(NEXTARG,0,10);
+ assert(!*++argv);
+
+ if (toolstack_save_fd >= 0)
+ helper_save_callbacks.toolstack_save = toolstack_save_cb;
+
+ helper_setcallbacks_save(&helper_save_callbacks, cbflags);
+
+ startup("save");
+ r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
+ &helper_save_callbacks, hvm, genidad);
+ complete(r);
+
+ } else if (!strcmp(mode,"--restore-domain")) {
+
+ int io_fd = atoi(NEXTARG);
+ uint32_t dom = strtoul(NEXTARG,0,10);
+ unsigned store_evtchn = strtoul(NEXTARG,0,10);
+ domid_t store_domid = strtoul(NEXTARG,0,10);
+ unsigned console_evtchn = strtoul(NEXTARG,0,10);
+ domid_t console_domid = strtoul(NEXTARG,0,10);
+ unsigned int hvm = strtoul(NEXTARG,0,10);
+ unsigned int pae = strtoul(NEXTARG,0,10);
+ int superpages = strtoul(NEXTARG,0,10);
+ int no_incr_genidad = strtoul(NEXTARG,0,10);
+ unsigned cbflags = strtoul(NEXTARG,0,10);
+ assert(!*++argv);
+
+ helper_setcallbacks_restore(&helper_restore_callbacks, cbflags);
+
+ unsigned long store_mfn = 0;
+ unsigned long console_mfn = 0;
+ unsigned long genidad = 0;
+
+ startup("restore");
+ r = xc_domain_restore(xch, io_fd, dom, store_evtchn, &store_mfn,
+ store_domid, console_evtchn, &console_mfn,
+ console_domid, hvm, pae, superpages,
+ no_incr_genidad, &genidad,
+ &helper_restore_callbacks);
+ helper_stub_restore_results(store_mfn,console_mfn,genidad,0);
+ complete(r);
+
+ } else {
+ assert(!"unexpected mode argument");
+ }
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_save_msgs_gen.pl
b/tools/libxl/libxl_save_msgs_gen.pl
new file mode 100755
index 0000000..c45986e
--- /dev/null
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -0,0 +1,397 @@
+#!/usr/bin/perl -w
+
+use warnings;
+use strict;
+use POSIX;
+
+our $debug = 0; # produce copious debugging output at run-time?
+
+our @msgs = (
+ # flags:
+ # s - applicable to save
+ # r - applicable to restore
+ # c - function pointer in callbacks struct rather than fixed function
+ # x - function pointer is in struct {save,restore}_callbacks
+ # and its null-ness needs to be passed through to the
helper''s xc
+ # W - needs a return value; callback is synchronous
+ [ 1, ''sr'', "log",
[qw(uint32_t level
+ uint32_t errnoval
+ STRING context
+ STRING formatted)] ],
+ [ 2, ''sr'', "progress",
[qw(STRING context
+ STRING doing_what),
+ ''unsigned
long'', ''done'',
+ ''unsigned
long'', ''total''] ],
+ [ 3, ''scxW'', "suspend", [] ],
+ [ 4, ''scxW'', "postcopy", [] ],
+ [ 5, ''scxW'', "checkpoint", [] ],
+ [ 6, ''scxW'', "switch_qemu_logdirty",
[qw(int domid
+ unsigned enable)] ],
+ # toolstack_save done entirely `by hand''
+ [ 7, ''rcxW'', "toolstack_restore",
[qw(uint32_t domid
+ BLOCK tsdata)] ],
+ [ 8, ''r'', "restore_results",
[''unsigned long'', ''store_mfn'',
+ ''unsigned
long'', ''console_mfn'',
+ ''unsigned
long'', ''genidad''] ],
+ [ 9, ''srW'', "complete",
[qw(int retval
+ int errnoval)] ],
+);
+
+#----------------------------------------
+
+our %cbs;
+our %func;
+our %func_ah;
+our @outfuncs;
+our %out_decls;
+our %out_body;
+our %msgnum_used;
+
+die unless @ARGV==1;
+die if $ARGV[0] =~ m/^-/;
+
+our ($intendedout) = @ARGV;
+
+$intendedout =~ m/([a-z]+)\.([ch])$/ or die;
+my ($want_ah, $ch) = ($1, $2);
+
+my $declprefix = '''';
+
+foreach my $ah (qw(callout helper)) {
+ $out_body{$ah} .+ <<END_BOTH.($ah eq
''callout'' ? <<END_CALLOUT : <<END_HELPER);
+#include "libxl_osdeps.h"
+
+#include <assert.h>
+#include <string.h>
+#include <stdint.h>
+#include <limits.h>
+END_BOTH
+
+#include "libxl_internal.h"
+
+END_CALLOUT
+
+#include "_libxl_save_msgs_${ah}.h"
+#include <xenctrl.h>
+#include <xenguest.h>
+
+END_HELPER
+}
+
+die $want_ah unless defined $out_body{$want_ah};
+
+sub f_decl ($$$$) {
+ my ($name, $ah, $c_rtype, $c_decl) = @_;
+ $out_decls{$name} = "${declprefix}$c_rtype $name$c_decl;\n";
+ $func{$name} = "$c_rtype $name$c_decl\n{\n" . ($func{$name} ||
'''');
+ $func_ah{$name} = $ah;
+}
+
+sub f_more ($$) {
+ my ($name, $addbody) = @_;
+ $func{$name} ||= '''';
+ $func{$name} .= $addbody;
+ push @outfuncs, $name;
+}
+
+our $libxl = "libxl__srm";
+our $callback = "${libxl}_callout_callback";
+our $receiveds = "${libxl}_callout_received";
+our $sendreply = "${libxl}_callout_sendreply";
+our $getcallbacks = "${libxl}_callout_get_callbacks";
+our $enumcallbacks = "${libxl}_callout_enumcallbacks";
+sub cbtype ($) { "${libxl}_".$_[0]."_autogen_callbacks"; };
+
+f_decl($sendreply, ''callout'', ''void'',
"(int r, void *user)");
+
+our $helper = "helper";
+our $encode = "${helper}_stub";
+our $allocbuf = "${helper}_allocbuf";
+our $transmit = "${helper}_transmitmsg";
+our $getreply = "${helper}_getreply";
+our $setcallbacks = "${helper}_setcallbacks";
+
+f_decl($allocbuf, ''helper'', ''unsigned char
*'', ''(int len, void *user)'');
+f_decl($transmit, ''helper'', ''void'',
+ ''(unsigned char *msg_freed, int len, void *user)'');
+f_decl($getreply, ''helper'', ''int'',
''(void *user)'');
+
+sub typeid ($) { my ($t) = @_; $t =~ s/\W/_/; return $t; };
+
+$out_body{''callout''} .= <<END;
+static int bytes_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ void *result, int rlen)
+{
+ if (endmsg - *msg < rlen) return 0;
+ memcpy(result,*msg,rlen);
+ *msg += rlen;
+ return 1;
+}
+
+END
+$out_body{''helper''} .= <<END;
+static void bytes_put(unsigned char *const buf, int *len,
+ const void *value, int vlen)
+{
+ assert(vlen < INT_MAX/2 - *len);
+ if (buf)
+ memcpy(buf + *len, value, vlen);
+ *len += vlen;
+}
+
+END
+
+foreach my $simpletype (qw(int uint16_t uint32_t unsigned), ''unsigned
long'') {
+ my $typeid = typeid($simpletype);
+ $out_body{''callout''} .= <<END;
+static int ${typeid}_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ $simpletype *result)
+{
+ return bytes_get(msg, endmsg, result, sizeof(*result));
+}
+
+END
+ $out_body{''helper''} .= <<END;
+static void ${typeid}_put(unsigned char *const buf, int *len,
+ const $simpletype value)
+{
+ bytes_put(buf, len, &value, sizeof(value));
+}
+
+END
+}
+
+$out_body{''callout''} .= <<END;
+static int BLOCK_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ const uint8_t **result, uint32_t *result_size)
+{
+ if (!uint32_t_get(msg,endmsg,result_size)) return 0;
+ if (endmsg - *msg < *result_size) return 0;
+ *result = (const void*)*msg;
+ *msg += *result_size;
+ return 1;
+}
+
+static int STRING_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ const char **result)
+{
+ const uint8_t *data;
+ uint32_t datalen;
+ if (!BLOCK_get(msg,endmsg,&data,&datalen)) return 0;
+ if (datalen == 0) return 0;
+ if (data[datalen-1] != ''\\0'') return 0;
+ *result = (const void*)data;
+ return 1;
+}
+
+END
+$out_body{''helper''} .= <<END;
+static void BLOCK_put(unsigned char *const buf,
+ int *len,
+ const uint8_t *bytes, uint32_t size)
+{
+ uint32_t_put(buf, len, size);
+ bytes_put(buf, len, bytes, size);
+}
+
+static void STRING_put(unsigned char *const buf,
+ int *len,
+ const char *string)
+{
+ size_t slen = strlen(string);
+ assert(slen < INT_MAX / 4);
+ assert(slen < (uint32_t)0x40000000);
+ BLOCK_put(buf, len, (const void*)string, slen+1);
+}
+
+END
+
+foreach my $sr (qw(save restore)) {
+ f_decl("${getcallbacks}_${sr}", ''callout'',
+ "const ".cbtype($sr)." *",
+ "(void *data)");
+
+ f_decl("${receiveds}_${sr}", ''callout'',
''int'',
+ "(const unsigned char *msg, uint32_t len, void *user)");
+
+ f_decl("${enumcallbacks}_${sr}", ''callout'',
''unsigned'',
+ "(const ".cbtype($sr)." *cbs)");
+ f_more("${enumcallbacks}_${sr}", " unsigned cbflags =
0;\n");
+
+ f_decl("${setcallbacks}_${sr}", ''helper'',
''void'',
+ "(struct ${sr}_callbacks *cbs, unsigned cbflags)");
+
+ f_more("${receiveds}_${sr}",
+ <<END_ALWAYS.($debug ? <<END_DEBUG :
'''').<<END_ALWAYS);
+ const unsigned char *const endmsg = msg + len;
+ uint16_t mtype;
+ if (!uint16_t_get(&msg,endmsg,&mtype)) return 0;
+END_ALWAYS
+ fprintf(stderr,"libxl callout receiver: got len=%u
mtype=%u\\n",len,mtype);
+END_DEBUG
+ switch (mtype) {
+
+END_ALWAYS
+
+ $cbs{$sr} = "typedef struct ".cbtype($sr)." {\n";
+}
+
+foreach my $msginfo (@msgs) {
+ my ($msgnum, $flags, $name, $args) = @$msginfo;
+ die if $msgnum_used{$msgnum}++;
+
+ my $f_more_sr = sub {
+ my ($contents_spec, $fnamebase) = @_;
+ $fnamebase ||= "${receiveds}";
+ foreach my $sr (qw(save restore)) {
+ $sr =~ m/^./;
+ next unless $flags =~ m/$&/;
+ my $contents = (!ref $contents_spec) ? $contents_spec :
+ $contents_spec->($sr);
+ f_more("${fnamebase}_${sr}", $contents);
+ }
+ };
+
+ $f_more_sr->(" case $msgnum: { /* $name */\n");
+ if ($flags =~ m/W/) {
+ $f_more_sr->(" int r;\n");
+ }
+
+ my $c_rtype_helper = $flags =~ m/W/ ? ''int'' :
''void'';
+ my $c_rtype_callout = $flags =~ m/W/ ? ''int'' :
''void'';
+ my $c_decl = ''('';
+ my $c_callback_args = '''';
+
+ f_more("${encode}_$name",
+ <<END_ALWAYS.($debug ? <<END_DEBUG :
'''').<<END_ALWAYS);
+ unsigned char *buf = 0;
+ int len = 0, allocd = 0;
+
+END_ALWAYS
+ fprintf(stderr,"libxl-save-helper: encoding $name\\n");
+END_DEBUG
+ for (;;) {
+ uint16_t_put(buf, &len, $msgnum /* $name */);
+END_ALWAYS
+
+ my @args = @$args;
+ my $c_recv = '''';
+ my ($argtype, $arg);
+ while (($argtype, $arg, @args) = @args) {
+ my $typeid = typeid($argtype);
+ my $c_args = "$arg";
+ my $c_get_args = "&$arg";
+ if ($argtype eq ''STRING'') {
+ $c_decl .= "const char *$arg, ";
+ $f_more_sr->(" const char *$arg;\n");
+ } elsif ($argtype eq ''BLOCK'') {
+ $c_decl .= "const uint8_t *$arg, uint32_t ${arg}_size, ";
+ $c_args .= ", ${arg}_size";
+ $c_get_args .= ",&${arg}_size";
+ $f_more_sr->(" const uint8_t *$arg;\n".
+ " uint32_t ${arg}_size;\n");
+ } else {
+ $c_decl .= "$argtype $arg, ";
+ $f_more_sr->(" $argtype $arg;\n");
+ }
+ $c_callback_args .= "$c_args, ";
+ $c_recv.+ " if
(!${typeid}_get(&msg,endmsg,$c_get_args)) return 0;\n";
+ f_more("${encode}_$name", " ${typeid}_put(buf, &len,
$c_args);\n");
+ }
+ $f_more_sr->($c_recv);
+ $c_decl .= "void *user)";
+ $c_callback_args .= "user";
+
+ $f_more_sr->(" if (msg != endmsg) return 0;\n");
+
+ my $c_callback;
+ if ($flags !~ m/c/) {
+ $c_callback = "${callback}_$name";
+ } else {
+ $f_more_sr->(sub {
+ my ($sr) = @_;
+ $cbs{$sr} .= " $c_rtype_callout (*${name})$c_decl;\n";
+ return
+ " const ".cbtype($sr)." *const cbs =\n".
+ " ${getcallbacks}_${sr}(user);\n";
+ });
+ $c_callback = "cbs->${name}";
+ }
+ my $c_make_callback = "$c_callback($c_callback_args)";
+ if ($flags !~ m/W/) {
+ $f_more_sr->(" $c_make_callback;\n");
+ } else {
+ $f_more_sr->(" r = $c_make_callback;\n".
+ " $sendreply(r, user);\n");
+ f_decl($sendreply, ''callout'', ''void'',
''(int r, void *user)'');
+ }
+ if ($flags =~ m/x/) {
+ my $c_v = "(1u<<$msgnum)";
+ my $c_cb = "cbs->$name";
+ $f_more_sr->(" if ($c_cb) cbflags |= $c_v;\n",
$enumcallbacks);
+ $f_more_sr->(" $c_cb = (cbflags & $c_v) ?
${encode}_${name} : 0;\n",
+ $setcallbacks);
+ }
+ $f_more_sr->(" return 1;\n }\n\n");
+ f_decl("${callback}_$name", ''callout'',
$c_rtype_callout, $c_decl);
+ f_decl("${encode}_$name", ''helper'',
$c_rtype_helper, $c_decl);
+ f_more("${encode}_$name",
+" if (buf) break;
+ buf = ${helper}_allocbuf(len, user);
+ assert(buf);
+ allocd = len;
+ len = 0;
+ }
+ assert(len == allocd);
+ ${transmit}(buf, len, user);
+");
+ if ($flags =~ m/W/) {
+ f_more("${encode}_$name",
+ (<<END_ALWAYS.($debug ? <<END_DEBUG :
'''').<<END_ALWAYS));
+ int r = ${helper}_getreply(user);
+END_ALWAYS
+ fprintf(stderr,"libxl-save-helper: $name got reply %d\\n",r);
+END_DEBUG
+ return r;
+END_ALWAYS
+ }
+}
+
+print "/* AUTOGENERATED by $0 DO NOT EDIT */\n\n" or die $!;
+
+foreach my $sr (qw(save restore)) {
+ f_more("${enumcallbacks}_${sr}",
+ " return cbflags;\n");
+ f_more("${receiveds}_${sr}",
+ " default:\n".
+ " return 0;\n".
+ " }");
+ $cbs{$sr} .= "} ".cbtype($sr).";\n\n";
+ if ($ch eq ''h'') {
+ print $cbs{$sr} or die $!;
+ print "struct ${sr}_callbacks;\n";
+ }
+}
+
+if ($ch eq ''c'') {
+ foreach my $name (@outfuncs) {
+ next unless defined $func{$name};
+ $func{$name} .= "}\n\n";
+ $out_body{$func_ah{$name}} .= $func{$name};
+ delete $func{$name};
+ }
+ print $out_body{$want_ah} or die $!;
+} else {
+ foreach my $name (sort keys %out_decls) {
+ next unless $func_ah{$name} eq $want_ah;
+ print $out_decls{$name} or die $!;
+ }
+}
+
+close STDOUT or die $!;
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 07/21] libxl: rename libxl_dom:save_helper to physmap_path
"save_helper" isn''t very descriptive. Also it is now
confusing
because it reads like it might refer to the libxl-save-helper
executable which runs xc_domain_save and xc_domain_restore.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl_dom.c | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index fea0c94..a597627 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -734,7 +734,7 @@ int libxl__domain_suspend_common_callback(void *user)
return 1;
}
-static inline char *save_helper(libxl__gc *gc, uint32_t domid,
+static inline char *physmap_path(libxl__gc *gc, uint32_t domid,
char *phys_offset, char *node)
{
return libxl__sprintf(gc,
@@ -779,21 +779,21 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
return -1;
}
- xs_path = save_helper(gc, domid, phys_offset, "start_addr");
+ xs_path = physmap_path(gc, domid, phys_offset, "start_addr");
start_addr = libxl__xs_read(gc, 0, xs_path);
if (start_addr == NULL) {
LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
- xs_path = save_helper(gc, domid, phys_offset, "size");
+ xs_path = physmap_path(gc, domid, phys_offset, "size");
size = libxl__xs_read(gc, 0, xs_path);
if (size == NULL) {
LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
- xs_path = save_helper(gc, domid, phys_offset, "name");
+ xs_path = physmap_path(gc, domid, phys_offset, "name");
name = libxl__xs_read(gc, 0, xs_path);
if (name == NULL)
namelen = 0;
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 08/21] libxl: provide libxl__xs_*_checked and libxl__xs_transaction_*
These useful utility functions make dealing with xenstore a little
less painful.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in v4:
* Fixed typo `commited'' in comment.
Changes in v3:
* Fixed typo `transacton'' in log messages.
---
tools/libxl/libxl_internal.h | 38 +++++++++++++++++++++
tools/libxl/libxl_xshelp.c | 76 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 114 insertions(+), 0 deletions(-)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 1a7b526..b108d00 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -498,6 +498,44 @@ _hidden bool libxl__xs_mkdir(libxl__gc *gc,
xs_transaction_t t,
_hidden char *libxl__xs_libxl_path(libxl__gc *gc, uint32_t domid);
+
+/*----- "checked" xenstore access functions -----*/
+/* Each of these functions will check that it succeeded; if it
+ * fails it logs and returns ERROR_FAIL.
+ */
+
+/* On success, *result_out came from the gc.
+ * On error, *result_out is undefined.
+ * ENOENT counts as success but sets *result_out=0
+ */
+int libxl__xs_read_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char **result_out);
+
+/* Does not include a trailing null.
+ * May usefully be combined with GCSPRINTF if the format string
+ * behaviour of libxl__xs_write is desirable. */
+int libxl__xs_write_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char *string);
+
+/* ENOENT is not an error (even if the parent directories don''t exist)
*/
+int libxl__xs_rm_checked(libxl__gc *gc, xs_transaction_t t, const char *path);
+
+/* Transaction functions, best used together.
+ * The caller should initialise *t to 0 (XBT_NULL) before calling start.
+ * Each function leaves *t!=0 iff the transaction needs cleaning up.
+ *
+ * libxl__xs_transaction_commit returns:
+ * <0 failure - a libxl error code
+ * +1 commit conflict; transaction has been destroyed and caller
+ * must go round again (call _start again and retry)
+ * 0 committed successfully
+ */
+int libxl__xs_transaction_start(libxl__gc *gc, xs_transaction_t *t);
+int libxl__xs_transaction_commit(libxl__gc *gc, xs_transaction_t *t);
+void libxl__xs_transaction_abort(libxl__gc *gc, xs_transaction_t *t);
+
+
+
/*
* This is a recursive delete, from top to bottom. What this function does
* is remove empty folders that contained the deleted entry.
diff --git a/tools/libxl/libxl_xshelp.c b/tools/libxl/libxl_xshelp.c
index c5b5364..993f527 100644
--- a/tools/libxl/libxl_xshelp.c
+++ b/tools/libxl/libxl_xshelp.c
@@ -135,6 +135,82 @@ char *libxl__xs_libxl_path(libxl__gc *gc, uint32_t domid)
return s;
}
+int libxl__xs_read_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char **result_out)
+{
+ char *result = libxl__xs_read(gc, t, path);
+ if (!result) {
+ if (errno != ENOENT) {
+ LOGE(ERROR, "xenstore read failed: `%s''", path);
+ return ERROR_FAIL;
+ }
+ }
+ *result_out = result;
+ return 0;
+}
+
+int libxl__xs_write_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char *string)
+{
+ size_t length = strlen(string);
+ if (!xs_write(CTX->xsh, t, path, string, length)) {
+ LOGE(ERROR, "xenstore write failed: `%s'' =
`%s''", path, string);
+ return ERROR_FAIL;
+ }
+ return 0;
+}
+
+int libxl__xs_rm_checked(libxl__gc *gc, xs_transaction_t t, const char *path)
+{
+ if (!xs_rm(CTX->xsh, t, path)) {
+ if (errno == ENOENT)
+ return 0;
+
+ LOGE(ERROR, "xenstore rm failed: `%s''", path);
+ return ERROR_FAIL;
+ }
+ return 0;
+}
+
+int libxl__xs_transaction_start(libxl__gc *gc, xs_transaction_t *t)
+{
+ assert(!*t);
+ *t = xs_transaction_start(CTX->xsh);
+ if (!*t) {
+ LOGE(ERROR, "could not create xenstore transaction");
+ return ERROR_FAIL;
+ }
+ return 0;
+}
+
+int libxl__xs_transaction_commit(libxl__gc *gc, xs_transaction_t *t)
+{
+ assert(*t);
+
+ if (!xs_transaction_end(CTX->xsh, *t, 0)) {
+ if (errno == EAGAIN)
+ return +1;
+
+ *t = 0;
+ LOGE(ERROR, "could not commit xenstore transaction");
+ return ERROR_FAIL;
+ }
+
+ *t = 0;
+ return 0;
+}
+
+void libxl__xs_transaction_abort(libxl__gc *gc, xs_transaction_t *t)
+{
+ if (!*t)
+ return;
+
+ if (!xs_transaction_end(CTX->xsh, *t, 1))
+ LOGE(ERROR, "could not abort xenstore transaction");
+
+ *t = 0;
+}
+
int libxl__xs_path_cleanup(libxl__gc *gc, xs_transaction_t t, char *user_path)
{
unsigned int nb = 0;
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 09/21] libxl: wait for qemu to acknowledge logdirty command
The current migration code in libxl instructs qemu to start or stop
logdirty, but it does not wait for an acknowledgement from qemu before
continuing. This might lead to memory corruption (!)
Fix this by waiting for qemu to acknowledge the command.
Unfortunately the necessary ao arrangements for waiting for this
command are unique because qemu has a special protocol for this
particular operation.
Also, this change means that the switch_qemu_logdirty callback
implementation in libxl can no longer synchronously produce its return
value, as it now needs to wait for xenstore. So we tell the
marshalling code generator that it is a message which does not need a
reply. This turns the callback function called by the marshaller into
one which returns void; the callback function arranges to later
explicitly sends the reply to the helper, when the xs watch triggers
and the appropriate value is read from xenstore.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in v4:
* Correct the sense of the return value from switch_qemu_logdirty.
* Fix a somewhat mendacious log message.
---
tools/libxl/libxl_dom.c | 177 +++++++++++++++++++++++++++++++++---
tools/libxl/libxl_internal.h | 18 ++++-
tools/libxl/libxl_save_callout.c | 8 ++
tools/libxl/libxl_save_msgs_gen.pl | 7 +-
4 files changed, 194 insertions(+), 16 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index a597627..4cca846 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -528,30 +528,181 @@ int libxl__toolstack_restore(uint32_t domid, const
uint8_t *buf,
static void domain_suspend_done(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc);
-/*----- callbacks, called by xc_domain_save -----*/
+/*----- complicated callback, called by xc_domain_save -----*/
+
+/*
+ * We implement the other end of protocol for controlling qemu-dm''s
+ * logdirty. There is no documentation for this protocol, but our
+ * counterparty''s implementation is in
+ * qemu-xen-traditional.git:xenstore.c in the function
+ * xenstore_process_logdirty_event
+ */
+
+static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
+ const struct timeval *requested_abs);
+static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*,
+ const char *watch_path, const char *event_path);
+static void switch_logdirty_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int ok);
-int libxl__domain_suspend_common_switch_qemu_logdirty
+static void logdirty_init(libxl__logdirty_switch *lds)
+{
+ lds->cmd_path = 0;
+ libxl__ev_xswatch_init(&lds->watch);
+ libxl__ev_time_init(&lds->timeout);
+}
+
+void libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned enable, void *user)
{
libxl__save_helper_state *shs = user;
+ libxl__egc *egc = shs->egc;
libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__logdirty_switch *lds = &dss->logdirty;
STATE_AO_GC(dss->ao);
- char *path;
- bool rc;
+ int rc;
+ xs_transaction_t t = 0;
+ const char *got;
- path = libxl__sprintf(gc,
+ if (!lds->cmd_path) {
+ lds->cmd_path = GCSPRINTF(
"/local/domain/0/device-model/%u/logdirty/cmd",
domid);
- if (!path)
- return 1;
+ lds->ret_path = GCSPRINTF(
+ "/local/domain/0/device-model/%u/logdirty/ret",
domid);
+ }
+ lds->cmd = enable ? "enable" : "disable";
- if (enable)
- rc = xs_write(CTX->xsh, XBT_NULL, path, "enable",
strlen("enable"));
- else
- rc = xs_write(CTX->xsh, XBT_NULL, path, "disable",
strlen("disable"));
+ rc = libxl__ev_xswatch_register(gc, &lds->watch,
+ switch_logdirty_xswatch, lds->ret_path);
+ if (rc) goto out;
+
+ rc = libxl__ev_time_register_rel(gc, &lds->timeout,
+ switch_logdirty_timeout, 10*1000);
+ if (rc) goto out;
+
+ for (;;) {
+ rc = libxl__xs_transaction_start(gc, &t);
+ if (rc) goto out;
+
+ rc = libxl__xs_read_checked(gc, t, lds->cmd_path, &got);
+ if (rc) goto out;
+
+ if (got) {
+ const char *got_ret;
+ rc = libxl__xs_read_checked(gc, t, lds->ret_path, &got_ret);
+ if (rc) goto out;
- return rc ? 0 : 1;
+ if (strcmp(got, got_ret)) {
+ LOG(ERROR,"controlling logdirty: qemu was already
sent"
+ " command `%s'' (xenstore path `%s'')
but result is `%s''",
+ got, lds->cmd_path, got_ret ? got_ret :
"<none>");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ rc = libxl__xs_rm_checked(gc, t, lds->cmd_path);
+ if (rc) goto out;
+ }
+
+ rc = libxl__xs_rm_checked(gc, t, lds->ret_path);
+ if (rc) goto out;
+
+ rc = libxl__xs_write_checked(gc, t, lds->cmd_path, lds->cmd);
+ if (rc) goto out;
+
+ rc = libxl__xs_transaction_commit(gc, &t);
+ if (!rc) break;
+ if (rc<0) goto out;
+ }
+
+ /* OK, wait for some callback */
+ return;
+
+ out:
+ LOG(ERROR,"logdirty switch failed (rc=%d), aborting suspend",rc);
+ switch_logdirty_done(egc,dss,-1);
+}
+
+static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
+ const struct timeval *requested_abs)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(ev, *dss,
logdirty.timeout);
+ STATE_AO_GC(dss->ao);
+ LOG(ERROR,"logdirty switch: wait for device model timed out");
+ switch_logdirty_done(egc,dss,-1);
}
+static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch,
+ const char *watch_path, const char *event_path)
+{
+ libxl__domain_suspend_state *dss + CONTAINER_OF(watch, *dss,
logdirty.watch);
+ libxl__logdirty_switch *lds = &dss->logdirty;
+ STATE_AO_GC(dss->ao);
+ const char *got;
+ xs_transaction_t t = 0;
+ int rc;
+
+ for (;;) {
+ rc = libxl__xs_transaction_start(gc, &t);
+ if (rc) goto out;
+
+ rc = libxl__xs_read_checked(gc, t, lds->ret_path, &got);
+ if (rc) goto out;
+
+ if (!got) {
+ rc = +1;
+ goto out;
+ }
+
+ if (strcmp(got, lds->cmd)) {
+ LOG(ERROR,"logdirty switch: sent command `%s'' but got
reply `%s''"
+ " (xenstore paths `%s'' / `%s'')",
lds->cmd, got,
+ lds->cmd_path, lds->ret_path);
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ rc = libxl__xs_rm_checked(gc, t, lds->cmd_path);
+ if (rc) goto out;
+
+ rc = libxl__xs_rm_checked(gc, t, lds->ret_path);
+ if (rc) goto out;
+
+ rc = libxl__xs_transaction_commit(gc, &t);
+ if (!rc) break;
+ if (rc<0) goto out;
+ }
+
+ out:
+ /* rc < 0: error
+ * rc == 0: ok, we are done
+ * rc == +1: need to keep waiting
+ */
+ libxl__xs_transaction_abort(gc, &t);
+
+ if (!rc) {
+ switch_logdirty_done(egc,dss,0);
+ } else if (rc < 0) {
+ LOG(ERROR,"logdirty switch: failed (rc=%d)",rc);
+ switch_logdirty_done(egc,dss,-1);
+ }
+}
+
+static void switch_logdirty_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ int broke)
+{
+ STATE_AO_GC(dss->ao);
+ libxl__logdirty_switch *lds = &dss->logdirty;
+
+ libxl__ev_xswatch_deregister(gc, &lds->watch);
+ libxl__ev_time_deregister(gc, &lds->timeout);
+
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs,
broke);
+}
+
+/*----- callbacks, called by xc_domain_save -----*/
+
int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid)
{
libxl_ctx *ctx = libxl__gc_owner(gc);
@@ -873,6 +1024,8 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
libxl__srm_save_autogen_callbacks *const callbacks
&dss->shs.callbacks.save.a;
+ logdirty_init(&dss->logdirty);
+
switch (type) {
case LIBXL_DOMAIN_TYPE_HVM: {
char *path;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index b108d00..05bed01 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1864,6 +1864,14 @@ typedef struct libxl__domain_suspend_state
libxl__domain_suspend_state;
typedef void libxl__domain_suspend_cb(libxl__egc*,
libxl__domain_suspend_state*, int rc);
+typedef struct libxl__logdirty_switch {
+ const char *cmd;
+ const char *cmd_path;
+ const char *ret_path;
+ libxl__ev_xswatch watch;
+ libxl__ev_time timeout;
+} libxl__logdirty_switch;
+
struct libxl__domain_suspend_state {
/* set by caller of libxl__domain_suspend */
libxl__ao *ao;
@@ -1883,6 +1891,7 @@ struct libxl__domain_suspend_state {
int guest_responded;
int interval; /* checkpoint interval (for Remus) */
libxl__save_helper_state shs;
+ libxl__logdirty_switch logdirty;
};
@@ -2013,8 +2022,15 @@ _hidden void libxl__xc_domain_save(libxl__egc*,
libxl__domain_suspend_state*,
_hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
int rc, int retval, int errnoval);
+/* Used by asynchronous callbacks: ie ones which xc regards as
+ * returning a value, but which we want to handle asynchronously.
+ * Such functions'' actual callback function return void in libxl
+ * When they are ready to indicate completion, they call this. */
+void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
+ libxl__save_helper_state *shs, int return_value);
+
_hidden int libxl__domain_suspend_common_callback(void *data);
-_hidden int libxl__domain_suspend_common_switch_qemu_logdirty
+_hidden void libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data);
_hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
uint32_t *len, void *data);
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 007d61c..297854e 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -142,6 +142,14 @@ void libxl__xc_domain_save(libxl__egc *egc,
libxl__domain_suspend_state *dss,
}
+void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
+ libxl__save_helper_state *shs, int return_value)
+{
+ shs->egc = egc;
+ libxl__srm_callout_sendreply(return_value, shs);
+ shs->egc = 0;
+}
+
/*----- helper execution -----*/
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
diff --git a/tools/libxl/libxl_save_msgs_gen.pl
b/tools/libxl/libxl_save_msgs_gen.pl
index c45986e..a9ac808 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -14,6 +14,7 @@ our @msgs = (
# x - function pointer is in struct {save,restore}_callbacks
# and its null-ness needs to be passed through to the
helper''s xc
# W - needs a return value; callback is synchronous
+ # A - needs a return value; callback is asynchronous
[ 1, ''sr'', "log",
[qw(uint32_t level
uint32_t errnoval
STRING context
@@ -25,7 +26,7 @@ our @msgs = (
[ 3, ''scxW'', "suspend", [] ],
[ 4, ''scxW'', "postcopy", [] ],
[ 5, ''scxW'', "checkpoint", [] ],
- [ 6, ''scxW'', "switch_qemu_logdirty",
[qw(int domid
+ [ 6, ''scxA'', "switch_qemu_logdirty",
[qw(int domid
unsigned enable)] ],
# toolstack_save done entirely `by hand''
[ 7, ''rcxW'', "toolstack_restore",
[qw(uint32_t domid
@@ -262,7 +263,7 @@ foreach my $msginfo (@msgs) {
$f_more_sr->(" int r;\n");
}
- my $c_rtype_helper = $flags =~ m/W/ ? ''int'' :
''void'';
+ my $c_rtype_helper = $flags =~ m/[WA]/ ? ''int'' :
''void'';
my $c_rtype_callout = $flags =~ m/W/ ? ''int'' :
''void'';
my $c_decl = ''('';
my $c_callback_args = '''';
@@ -351,7 +352,7 @@ END_ALWAYS
assert(len == allocd);
${transmit}(buf, len, user);
");
- if ($flags =~ m/W/) {
+ if ($flags =~ m/[WA]/) {
f_more("${encode}_$name",
(<<END_ALWAYS.($debug ? <<END_DEBUG :
'''').<<END_ALWAYS));
int r = ${helper}_getreply(user);
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 10/21] libxl: datacopier: provide "prefix data" facility
This will be used to write the qemu data banner to the save/migration
stream.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in v3:
* Clarified and added comments explaining the `immediately''
constraint and the lack of a reentrancy/threading hazard.
* Fixed subject line typo.
---
tools/libxl/libxl_aoutils.c | 22 ++++++++++++++++++++++
tools/libxl/libxl_internal.h | 6 ++++++
2 files changed, 28 insertions(+), 0 deletions(-)
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index ee0df57..7f8d6d3 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -74,6 +74,28 @@ static void datacopier_check_state(libxl__egc *egc,
libxl__datacopier_state *dc)
}
}
+void libxl__datacopier_prefixdata(libxl__egc *egc, libxl__datacopier_state *dc,
+ const void *data, size_t len)
+{
+ libxl__datacopier_buf *buf;
+ /*
+ * It is safe for this to be called immediately after _start, as
+ * is documented in the public comment. _start''s caller must have
+ * the ctx locked, so other threads don''t get to mess with the
+ * contents, and the fd events cannot happen reentrantly. So we
+ * are guaranteed to beat the first data from the read fd.
+ */
+
+ assert(len < dc->maxsz - dc->used);
+
+ buf = libxl__zalloc(0, sizeof(*buf) - sizeof(buf->buf) + len);
+ buf->used = len;
+ memcpy(buf->buf, data, len);
+
+ dc->used += len;
+ LIBXL_TAILQ_INSERT_TAIL(&dc->bufs, buf, entry);
+}
+
static void datacopier_readable(libxl__egc *egc, libxl__ev_fd *ev,
int fd, short events, short revents) {
libxl__datacopier_state *dc = CONTAINER_OF(ev, *dc, toread);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 05bed01..8b582e4 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1811,6 +1811,12 @@ _hidden void
libxl__datacopier_init(libxl__datacopier_state *dc);
_hidden void libxl__datacopier_kill(libxl__datacopier_state *dc);
_hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
+/* Inserts literal data into the output stream. The data is copied.
+ * May safely be used only immediately after libxl__datacopier_start
+ * (before the ctx is unlocked). But may be called multiple times.
+ * NB exceeding maxsz will fail an assertion! */
+_hidden void libxl__datacopier_prefixdata(libxl__egc*,
libxl__datacopier_state*,
+ const void *data, size_t len);
/*----- Save/restore helper (used by creation and suspend) -----*/
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 11/21] libxl: prepare for asynchronous writing of qemu save file
* Combine the various calls to libxl__device_model_savefile into one
at the start of libxl__domain_suspend, storing the result in the
dss. Consequently a few functions take a dss instead of some or all
of their other arguments.
* Make libxl__domain_save_device_model''s API into an asynchronous
style which takes a callback. The function is, however, still
synchronous; it will be made actually async in the next patch.
* Consequently make libxl__remus_domain_checkpoint_callback into an
asynchronous callback.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl_dom.c | 54 ++++++++++++++++++++++++++----------
tools/libxl/libxl_internal.h | 18 ++++++++++--
tools/libxl/libxl_save_msgs_gen.pl | 2 +-
3 files changed, 55 insertions(+), 19 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 4cca846..e104744 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -703,11 +703,13 @@ static void switch_logdirty_done(libxl__egc *egc,
/*----- callbacks, called by xc_domain_save -----*/
-int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid)
+int libxl__domain_suspend_device_model(libxl__gc *gc,
+ libxl__domain_suspend_state *dss)
{
libxl_ctx *ctx = libxl__gc_owner(gc);
int ret = 0;
- const char *filename = libxl__device_model_savefile(gc, domid);
+ uint32_t const domid = dss->domid;
+ const char *const filename = dss->dm_savefile;
switch (libxl__device_model_version_running(gc, domid)) {
case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL: {
@@ -876,7 +878,7 @@ int libxl__domain_suspend_common_callback(void *user)
guest_suspended:
if (dss->hvm) {
- ret = libxl__domain_suspend_device_model(gc, dss->domid);
+ ret = libxl__domain_suspend_device_model(gc, dss);
if (ret) {
LOG(ERROR, "libxl__domain_suspend_device_model failed
ret=%d", ret);
return 0;
@@ -991,19 +993,32 @@ static int libxl__remus_domain_resume_callback(void *data)
return 1;
}
-static int libxl__remus_domain_checkpoint_callback(void *data)
+/*----- remus asynchronous checkpoint callback -----*/
+
+static void remus_checkpoint_dm_saved(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int
rc);
+
+static void libxl__remus_domain_checkpoint_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
+ libxl__egc *egc = dss->shs.egc;
STATE_AO_GC(dss->ao);
/* This would go into tailbuf. */
- if (dss->hvm &&
- libxl__domain_save_device_model(gc, dss->domid, dss->fd))
- return 0;
+ if (dss->hvm) {
+ libxl__domain_save_device_model(egc, dss, remus_checkpoint_dm_saved);
+ } else {
+ remus_checkpoint_dm_saved(egc, dss, 0);
+ }
+}
+static void remus_checkpoint_dm_saved(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
+{
/* TODO: Wait for disk and memory ack, release network buffer */
+ /* TODO: make this asynchronous */
usleep(dss->interval * 1000);
- return 1;
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
}
/*----- main code for suspending, in order of execution -----*/
@@ -1053,6 +1068,7 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
dss->suspend_eventchn = -1;
dss->guest_responded = 0;
+ dss->dm_savefile = libxl__device_model_savefile(gc, domid);
if (r_info != NULL) {
dss->interval = r_info->interval;
@@ -1102,7 +1118,6 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void
*dss_void,
/* Convenience aliases */
const libxl_domain_type type = dss->type;
- const uint32_t domid = dss->domid;
if (rc)
goto out;
@@ -1120,11 +1135,11 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void
*dss_void,
}
if (type == LIBXL_DOMAIN_TYPE_HVM) {
- rc = libxl__domain_suspend_device_model(gc, domid);
+ rc = libxl__domain_suspend_device_model(gc, dss);
if (rc) goto out;
- rc = libxl__domain_save_device_model(gc, domid, dss->fd);
- if (rc) goto out;
+ libxl__domain_save_device_model(egc, dss, domain_suspend_done);
+ return;
}
rc = 0;
@@ -1133,14 +1148,22 @@ out:
domain_suspend_done(egc, dss, rc);
}
-int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd)
+void libxl__domain_save_device_model(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ libxl__save_device_model_cb *callback)
{
+ STATE_AO_GC(dss->ao);
int rc, fd2 = -1, c;
char buf[1024];
- const char *filename = libxl__device_model_savefile(gc, domid);
struct stat st;
uint32_t qemu_state_len;
+ dss->save_dm_callback = callback;
+
+ /* Convenience aliases */
+ const char *const filename = dss->dm_savefile;
+ const int fd = dss->fd;
+
if (stat(filename, &st) < 0)
{
LOG(ERROR, "Unable to stat qemu save file\n");
@@ -1182,7 +1205,8 @@ int libxl__domain_save_device_model(libxl__gc *gc,
uint32_t domid, int fd)
out:
if (fd2 >= 0) close(fd2);
unlink(filename);
- return rc;
+
+ dss->save_dm_callback(egc, dss, rc);
}
static void domain_suspend_done(libxl__egc *egc,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8b582e4..e95892a 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -824,10 +824,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t
domid,
_hidden int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data);
-_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
-_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
-_hidden int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int
fd);
+
_hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid);
@@ -1869,6 +1867,8 @@ typedef struct libxl__domain_suspend_state
libxl__domain_suspend_state;
typedef void libxl__domain_suspend_cb(libxl__egc*,
libxl__domain_suspend_state*, int rc);
+typedef void libxl__save_device_model_cb(libxl__egc*,
+ libxl__domain_suspend_state*, int rc);
typedef struct libxl__logdirty_switch {
const char *cmd;
@@ -1895,9 +1895,12 @@ struct libxl__domain_suspend_state {
int hvm;
int xcflags;
int guest_responded;
+ const char *dm_savefile;
int interval; /* checkpoint interval (for Remus) */
libxl__save_helper_state shs;
libxl__logdirty_switch logdirty;
+ /* private for libxl__domain_save_device_model */
+ libxl__save_device_model_cb *save_dm_callback;
};
@@ -2053,6 +2056,15 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc,
_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int rc, int retval, int errnoval);
+/* Each time the dm needs to be saved, we must call suspend and then save */
+_hidden int libxl__domain_suspend_device_model(libxl__gc *gc,
+ libxl__domain_suspend_state *dss);
+_hidden void libxl__domain_save_device_model(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ libxl__save_device_model_cb *callback);
+
+_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
+
/*
* Convenience macros.
diff --git a/tools/libxl/libxl_save_msgs_gen.pl
b/tools/libxl/libxl_save_msgs_gen.pl
index a9ac808..ee126c7 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -25,7 +25,7 @@ our @msgs = (
''unsigned
long'', ''total''] ],
[ 3, ''scxW'', "suspend", [] ],
[ 4, ''scxW'', "postcopy", [] ],
- [ 5, ''scxW'', "checkpoint", [] ],
+ [ 5, ''scxA'', "checkpoint", [] ],
[ 6, ''scxA'', "switch_qemu_logdirty",
[qw(int domid
unsigned enable)] ],
# toolstack_save done entirely `by hand''
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 12/21] libxl: Make libxl__domain_save_device_model asynchronous
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in series v3:
* Improve one more a debugging message.
---
tools/libxl/libxl_dom.c | 100 +++++++++++++++++++++++++++---------------
tools/libxl/libxl_internal.h | 1 +
2 files changed, 66 insertions(+), 35 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index e104744..dbae98c 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1148,15 +1148,17 @@ out:
domain_suspend_done(egc, dss, rc);
}
+static void save_device_model_datacopier_done(libxl__egc *egc,
+ libxl__datacopier_state *dc, int onwrite, int errnoval);
+
void libxl__domain_save_device_model(libxl__egc *egc,
libxl__domain_suspend_state *dss,
libxl__save_device_model_cb *callback)
{
STATE_AO_GC(dss->ao);
- int rc, fd2 = -1, c;
- char buf[1024];
struct stat st;
uint32_t qemu_state_len;
+ int rc;
dss->save_dm_callback = callback;
@@ -1164,49 +1166,77 @@ void libxl__domain_save_device_model(libxl__egc *egc,
const char *const filename = dss->dm_savefile;
const int fd = dss->fd;
- if (stat(filename, &st) < 0)
+ libxl__datacopier_state *dc = &dss->save_dm_datacopier;
+ memset(dc, 0, sizeof(*dc));
+ dc->readwhat = GCSPRINTF("qemu save file %s", filename);
+ dc->ao = ao;
+ dc->readfd = -1;
+ dc->writefd = fd;
+ dc->maxsz = INT_MAX;
+ dc->copywhat = GCSPRINTF("qemu save file for domain %"PRIu32,
dss->domid);
+ dc->writewhat = "save/migration stream";
+ dc->callback = save_device_model_datacopier_done;
+
+ dc->readfd = open(filename, O_RDONLY);
+ if (dc->readfd < 0) {
+ LOGE(ERROR, "unable to open %s", dc->readwhat);
+ goto out;
+ }
+
+ if (fstat(dc->readfd, &st))
{
- LOG(ERROR, "Unable to stat qemu save file\n");
- rc = ERROR_FAIL;
+ LOGE(ERROR, "unable to fstat %s", dc->readwhat);
+ goto out;
+ }
+
+ if (!S_ISREG(st.st_mode)) {
+ LOG(ERROR, "%s is not a plain file!", dc->readwhat);
goto out;
}
qemu_state_len = st.st_size;
- LOG(DEBUG, "Qemu state is %d bytes\n", qemu_state_len);
+ LOG(DEBUG, "%s is %d bytes", dc->readwhat, qemu_state_len);
- rc = libxl_write_exactly(CTX, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE),
- "saved-state file", "qemu
signature");
- if (rc)
- goto out;
+ rc = libxl__datacopier_start(dc);
+ if (rc) goto out;
- rc = libxl_write_exactly(CTX, fd, &qemu_state_len,
sizeof(qemu_state_len),
- "saved-state file", "saved-state
length");
- if (rc)
- goto out;
+ libxl__datacopier_prefixdata(egc, dc,
+ QEMU_SIGNATURE, strlen(QEMU_SIGNATURE));
- fd2 = open(filename, O_RDONLY);
- if (fd2 < 0) {
- LOGE(ERROR, "Unable to open qemu save file\n");
- goto out;
- }
- while ((c = read(fd2, buf, sizeof(buf))) != 0) {
- if (c < 0) {
- if (errno == EINTR)
- continue;
- rc = errno;
- goto out;
- }
- rc = libxl_write_exactly(
- CTX, fd, buf, c, "saved-state file", "qemu
state");
- if (rc)
- goto out;
+ libxl__datacopier_prefixdata(egc, dc,
+ &qemu_state_len, sizeof(qemu_state_len));
+ return;
+
+ out:
+ save_device_model_datacopier_done(egc, dc, -1, 0);
+}
+
+static void save_device_model_datacopier_done(libxl__egc *egc,
+ libxl__datacopier_state *dc, int onwrite, int errnoval)
+{
+ libxl__domain_suspend_state *dss + CONTAINER_OF(dc, *dss,
save_dm_datacopier);
+ STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases */
+ const char *const filename = dss->dm_savefile;
+ int our_rc = 0;
+ int rc;
+
+ libxl__datacopier_kill(dc);
+
+ if (onwrite || errnoval)
+ our_rc = ERROR_FAIL;
+
+ if (dc->readfd >= 0) {
+ close(dc->readfd);
+ dc->readfd = -1;
}
- rc = 0;
-out:
- if (fd2 >= 0) close(fd2);
- unlink(filename);
- dss->save_dm_callback(egc, dss, rc);
+ rc = libxl__remove_file(gc, filename);
+ if (!our_rc) our_rc = rc;
+
+ dss->save_dm_callback(egc, dss, our_rc);
}
static void domain_suspend_done(libxl__egc *egc,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e95892a..c9b4189 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1901,6 +1901,7 @@ struct libxl__domain_suspend_state {
libxl__logdirty_switch logdirty;
/* private for libxl__domain_save_device_model */
libxl__save_device_model_cb *save_dm_callback;
+ libxl__datacopier_state save_dm_datacopier;
};
--
1.7.2.5
In the next patch we are going to change the definition of NOGC to
require a local variable libxl__gc *gc.
libxl_get_cpu_topology doesn''t have one but does use NOGC.
Fix this by:
- introducing an `out'' label
- replacing the only call to `return'' with a suitable assignment
to ret and a `goto out''.
- adding uses of GC_INIT and GC_FREE.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl.c | 6 +++++-
1 files changed, 5 insertions(+), 1 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index f7fcf1b..ce990ab 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -3201,6 +3201,7 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo
*physinfo)
libxl_cputopology *libxl_get_cpu_topology(libxl_ctx *ctx, int *nr)
{
+ GC_INIT(ctx);
xc_topologyinfo_t tinfo;
DECLARE_HYPERCALL_BUFFER(xc_cpu_to_core_t, coremap);
DECLARE_HYPERCALL_BUFFER(xc_cpu_to_socket_t, socketmap);
@@ -3213,7 +3214,8 @@ libxl_cputopology *libxl_get_cpu_topology(libxl_ctx *ctx,
int *nr)
if (max_cpus == 0)
{
LIBXL__LOG(ctx, XTL_ERROR, "Unable to determine number of
CPUS");
- return NULL;
+ ret = NULL;
+ goto out;
}
coremap = xc_hypercall_buffer_alloc
@@ -3258,6 +3260,8 @@ fail:
if (ret)
*nr = max_cpus;
+ out:
+ GC_FREE;
return ret;
}
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:53 UTC
[PATCH 14/21] libxl: Do not pass NULL as gc_opt; introduce NOGC
In 25182:6c3345d7e9d9 the practice of passing NULL to gc-using memory
allocation functions was introduced. However, the arrangements there
were not correct as committed, because the error handling and logging
depends on getting a ctx from the gc - so an allocation error would in
fact result in libxl dereferencing NULL.
Instead, provide a special dummy gc in the ctx, called `nogc_gc''. It
is marked out specially by having alloc_maxsize==-1, which is
otherwise invalid.
Functions which need to actually look into the gc use the new test
function gc_is_real (whose purpose is mainly clarity of the code) to
check whether the gc is the dummy one, and do nothing if it is. And
we provide a helper macro NOGC which uses the in-scope real gc to find
the ctx and hence the dummy gc (and which replaces the previous
#define NOGC NULL).
Change all callers which pass 0 or NULL to an allocation function to
use NOGC or &ctx->nogc_gc, as applicable in the context.
We add a comment near the definition of LIBXL_INIT_GC pointing out
that it isn''t any more the only place a libxl__gc struct is
initialised, for the benefit of anyone changing the contents of gc''s
in the future.
Also, actually document that libxl__ptr_add is legal with ptr==NULL,
and change a couple of calls not to check for NULL argument.
Reported-by: Bamvor Jian Zhang <bjzhang@suse.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Bamvor Jian Zhang <bjzhang@suse.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
---
tools/libxl/libxl.c | 3 +++
tools/libxl/libxl_aoutils.c | 3 ++-
tools/libxl/libxl_create.c | 2 +-
tools/libxl/libxl_event.c | 5 +++--
tools/libxl/libxl_exec.c | 2 +-
tools/libxl/libxl_fork.c | 2 +-
tools/libxl/libxl_internal.c | 11 +++++++++--
tools/libxl/libxl_internal.h | 37 +++++++++++++++++++++++--------------
tools/libxl/libxl_utils.c | 6 ++----
tools/libxl/libxl_xshelp.c | 7 ++-----
10 files changed, 47 insertions(+), 31 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index ce990ab..d3b017b 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -41,6 +41,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
/* First initialise pointers etc. (cannot fail) */
+ ctx->nogc_gc.alloc_maxsize = -1;
+ ctx->nogc_gc.owner = ctx;
+
LIBXL_TAILQ_INIT(&ctx->occurred);
ctx->osevent_hooks = 0;
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index 7f8d6d3..99972a2 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -77,6 +77,7 @@ static void datacopier_check_state(libxl__egc *egc,
libxl__datacopier_state *dc)
void libxl__datacopier_prefixdata(libxl__egc *egc, libxl__datacopier_state *dc,
const void *data, size_t len)
{
+ EGC_GC;
libxl__datacopier_buf *buf;
/*
* It is safe for this to be called immediately after _start, as
@@ -88,7 +89,7 @@ void libxl__datacopier_prefixdata(libxl__egc *egc,
libxl__datacopier_state *dc,
assert(len < dc->maxsz - dc->used);
- buf = libxl__zalloc(0, sizeof(*buf) - sizeof(buf->buf) + len);
+ buf = libxl__zalloc(NOGC, sizeof(*buf) - sizeof(buf->buf) + len);
buf->used = len;
memcpy(buf->buf, data, len);
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 00705d8..ab000bc 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -108,7 +108,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
}
if (b_info->blkdev_start == NULL)
- b_info->blkdev_start = libxl__strdup(0, "xvda");
+ b_info->blkdev_start = libxl__strdup(NOGC, "xvda");
if (b_info->type == LIBXL_DOMAIN_TYPE_HVM) {
if (!b_info->u.hvm.bios)
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 565d2c2..eb23a93 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -772,7 +772,7 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller
*poller,
if (poller->fd_rindices_allocd < maxfd) {
assert(ARRAY_SIZE_OK(poller->fd_rindices, maxfd));
poller->fd_rindices - libxl__realloc(0,
poller->fd_rindices,
+ libxl__realloc(NOGC, poller->fd_rindices,
maxfd * sizeof(*poller->fd_rindices));
memset(poller->fd_rindices + poller->fd_rindices_allocd,
0,
@@ -1099,9 +1099,10 @@ void libxl_event_free(libxl_ctx *ctx, libxl_event *event)
libxl_event *libxl__event_new(libxl__egc *egc,
libxl_event_type type, uint32_t domid)
{
+ EGC_GC;
libxl_event *ev;
- ev = libxl__zalloc(0,sizeof(*ev));
+ ev = libxl__zalloc(NOGC,sizeof(*ev));
ev->type = type;
ev->domid = domid;
diff --git a/tools/libxl/libxl_exec.c b/tools/libxl/libxl_exec.c
index 082bf44..cfa379c 100644
--- a/tools/libxl/libxl_exec.c
+++ b/tools/libxl/libxl_exec.c
@@ -280,7 +280,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
int status, rc;
libxl__spawn_init(ss);
- ss->ssd = libxl__zalloc(0, sizeof(*ss->ssd));
+ ss->ssd = libxl__zalloc(NOGC, sizeof(*ss->ssd));
libxl__ev_child_init(&ss->ssd->mid);
rc = libxl__ev_time_register_rel(gc, &ss->timeout,
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
index 9ff99e0..044ddad 100644
--- a/tools/libxl/libxl_fork.c
+++ b/tools/libxl/libxl_fork.c
@@ -92,7 +92,7 @@ libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
libxl__carefd *cf = 0;
libxl_fd_set_cloexec(ctx, fd, 1);
- cf = libxl__zalloc(NULL, sizeof(*cf));
+ cf = libxl__zalloc(&ctx->nogc_gc, sizeof(*cf));
cf->fd = fd;
LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry);
return cf;
diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
index 8139520..fbff7d0 100644
--- a/tools/libxl/libxl_internal.c
+++ b/tools/libxl/libxl_internal.c
@@ -30,11 +30,16 @@ void libxl__alloc_failed(libxl_ctx *ctx, const char *func,
#undef L
}
+static int gc_is_real(const libxl__gc *gc)
+{
+ return gc->alloc_maxsize >= 0;
+}
+
void libxl__ptr_add(libxl__gc *gc, void *ptr)
{
int i;
- if (!gc)
+ if (!gc_is_real(gc))
return;
if (!ptr)
@@ -66,6 +71,8 @@ void libxl__free_all(libxl__gc *gc)
void *ptr;
int i;
+ assert(gc_is_real(gc));
+
for (i = 0; i < gc->alloc_maxsize; i++) {
ptr = gc->alloc_ptrs[i];
gc->alloc_ptrs[i] = NULL;
@@ -104,7 +111,7 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t
new_size)
if (ptr == NULL) {
libxl__ptr_add(gc, new_ptr);
- } else if (new_ptr != ptr && gc != NULL) {
+ } else if (new_ptr != ptr && gc_is_real(gc)) {
for (i = 0; i < gc->alloc_maxsize; i++) {
if (gc->alloc_ptrs[i] == ptr) {
gc->alloc_ptrs[i] = new_ptr;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index c9b4189..aa150b5 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -277,10 +277,18 @@ struct libxl__poller {
int wakeup_pipe[2]; /* 0 means no fd allocated */
};
+struct libxl__gc {
+ /* mini-GC */
+ int alloc_maxsize; /* -1 means this is the dummy non-gc gc */
+ void **alloc_ptrs;
+ libxl_ctx *owner;
+};
+
struct libxl__ctx {
xentoollog_logger *lg;
xc_interface *xch;
struct xs_handle *xsh;
+ libxl__gc nogc_gc;
const libxl_event_hooks *event_hooks;
void *event_hooks_user;
@@ -356,13 +364,6 @@ typedef struct {
#define PRINTF_ATTRIBUTE(x, y) __attribute__((format(printf, x, y)))
-struct libxl__gc {
- /* mini-GC */
- int alloc_maxsize;
- void **alloc_ptrs;
- libxl_ctx *owner;
-};
-
struct libxl__egc {
/* For event-generating functions only.
* The egc and its gc may be accessed only on the creating thread. */
@@ -420,6 +421,7 @@ struct libxl__ao {
(gc).alloc_ptrs = 0; \
(gc).owner = (ctx); \
} while(0)
+ /* NB, also, a gc struct ctx->nogc_gc is initialised in libxl_ctx_alloc
*/
static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
{
@@ -438,13 +440,20 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
* All pointers returned by these functions are registered for garbage
* collection on exit from the outermost libxl callframe.
*
- * However, where the argument is stated to be "gc_opt", NULL may be
- * passed instead, in which case no garbage collection will occur; the
- * pointer must later be freed with free(). This is for memory
- * allocations of types (b) and (c).
+ * However, where the argument is stated to be "gc_opt",
&ctx->nogc_gc
+ * may be passed instead, in which case no garbage collection will
+ * occur; the pointer must later be freed with free(). (Passing NULL
+ * for gc_opt is not permitted.) This is for memory allocations of
+ * types (b) and (c). The convenience macro NOGC should be used where
+ * possible.
+ *
+ * NOGC (and ctx->nogc_gc) may ONLY be used with functions which
+ * explicitly declare that it''s OK. Use with nonconsenting functions
+ * may result in leaks of those functions'' internal allocations on the
+ * psuedo-gc.
*/
-/* register @ptr in @gc for free on exit from outermost libxl callframe. */
-_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr);
+/* register ptr in gc for free on exit from outermost libxl callframe. */
+_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */);
/* if this is the outermost libxl callframe then free all pointers in @gc */
_hidden void libxl__free_all(libxl__gc *gc);
/* allocate and zero @bytes. (similar to a gc''d malloc(3)+memzero())
*/
@@ -2110,7 +2119,7 @@ _hidden const char *libxl__device_model_savefile(libxl__gc
*gc, uint32_t domid);
#define GC_INIT(ctx) libxl__gc gc[1]; LIBXL_INIT_GC(gc[0],ctx)
#define GC_FREE libxl__free_all(gc)
#define CTX libxl__gc_owner(gc)
-#define NOGC NULL
+#define NOGC (&CTX->nogc_gc) /* pass only to consenting
functions */
/* Allocation macros all of which use the gc. */
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index 67ef82c..08c7dac 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -58,8 +58,7 @@ char *libxl_domid_to_name(libxl_ctx *ctx, uint32_t domid)
char *libxl__domid_to_name(libxl__gc *gc, uint32_t domid)
{
char *s = libxl_domid_to_name(libxl__gc_owner(gc), domid);
- if ( s )
- libxl__ptr_add(gc, s);
+ libxl__ptr_add(gc, s);
return s;
}
@@ -107,8 +106,7 @@ char *libxl_cpupoolid_to_name(libxl_ctx *ctx, uint32_t
poolid)
char *libxl__cpupoolid_to_name(libxl__gc *gc, uint32_t poolid)
{
char *s = libxl_cpupoolid_to_name(libxl__gc_owner(gc), poolid);
- if ( s )
- libxl__ptr_add(gc, s);
+ libxl__ptr_add(gc, s);
return s;
}
diff --git a/tools/libxl/libxl_xshelp.c b/tools/libxl/libxl_xshelp.c
index 993f527..7fdf164 100644
--- a/tools/libxl/libxl_xshelp.c
+++ b/tools/libxl/libxl_xshelp.c
@@ -86,11 +86,8 @@ char * libxl__xs_read(libxl__gc *gc, xs_transaction_t t,
const char *path)
char *ptr;
ptr = xs_read(ctx->xsh, t, path, NULL);
- if (ptr != NULL) {
- libxl__ptr_add(gc, ptr);
- return ptr;
- }
- return 0;
+ libxl__ptr_add(gc, ptr);
+ return ptr;
}
char *libxl__xs_get_dompath(libxl__gc *gc, uint32_t domid)
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:54 UTC
[PATCH 15/21] libxl: Get compiler to warn about gc_opt==NULL
Since it used to be legal to pass gc_opt==NULL, and there are various patches floating about and under development which do so, add a compiler annotation which makes the build fail when that is done. This turns a runtime crash into a build failure, and should ensure that we don''t accidentally commit a broken combination of patches. This is something of an annoying approach because it adds a macro invocation to the RHS of every declaration of a function taking a gc_opt. So it should be reverted after Xen 4.2rc1. Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> --- tools/libxl/libxl_internal.h | 21 +++++++++++++-------- 1 files changed, 13 insertions(+), 8 deletions(-) diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index aa150b5..85f4bc6 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -453,28 +453,33 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc) * psuedo-gc. */ /* register ptr in gc for free on exit from outermost libxl callframe. */ -_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */); + +#define NN1 __attribute__((nonnull(1))) + /* It used to be legal to pass NULL for gc_opt. Get the compiler to + * warn about this if any slip through. */ + +_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */) NN1; /* if this is the outermost libxl callframe then free all pointers in @gc */ _hidden void libxl__free_all(libxl__gc *gc); /* allocate and zero @bytes. (similar to a gc''d malloc(3)+memzero()) */ -_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes); +_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes) NN1; /* allocate and zero memory for an array of @nmemb members of @size each. * (similar to a gc''d calloc(3)). */ -_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size); +_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size) NN1; /* change the size of the memory block pointed to by @ptr to @new_size bytes. * unlike other allocation functions here any additional space between the * oldsize and @new_size is not initialised (similar to a gc''d realloc(3)). */ -_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size); +_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size) NN1; /* print @fmt into an allocated string large enoughto contain the result. * (similar to gc''d asprintf(3)). */ -_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3); +_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3) NN1; /* duplicate the string @c (similar to a gc''d strdup(3)). */ -_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c); +_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c) NN1; /* duplicate at most @n bytes of string @c (similar to a gc''d strndup(3)). */ -_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n); +_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n) NN1; /* strip the last path component from @s and return as a newly allocated * string. (similar to a gc''d dirname(3)). */ -_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s); +_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s) NN1; /* Each of these logs errors and returns a libxl error code. * They do not mind if path is already removed. -- 1.7.2.5
Ian Jackson
2012-Jun-15 11:54 UTC
[PATCH 16/21] xl: Handle return value from libxl_domain_suspend correctly
libxl_domain_suspend returns a libxl error code. So it must be
wrapped with MUST and not CHK_ERRNO.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/xl_cmdimpl.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 19daa1c..a9125cc 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -2846,7 +2846,7 @@ static int save_domain(const char *p, const char
*filename, int checkpoint,
save_domain_core_writeconfig(fd, filename, config_data, config_len);
- CHK_ERRNO(libxl_domain_suspend(ctx, domid, fd, 0, NULL));
+ MUST(libxl_domain_suspend(ctx, domid, fd, 0, NULL));
close(fd);
if (checkpoint)
--
1.7.2.5
This was allocated using asprintf but never freed. Use GCSPRINTF.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl_create.c | 3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index ab000bc..5618293 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -740,9 +740,8 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void
*dcs_void,
goto out;
if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
- ret = asprintf(&state->saved_state,
+ state->saved_state = GCSPRINTF(
XC_DEVICE_MODEL_RESTORE_FILE".%d", domid);
- ret = (ret < 0) ? ERROR_FAIL : 0;
}
out:
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:54 UTC
[PATCH 18/21] libxl: do not leak spawned middle children
libxl__spawn_spawn would, when libxl__spawn_detach was called, make
the spawn become idle immediately. However it still has a child
process which needs to be waited for: the `detachable'' spawned
child.
This is wrong because the ultimate in-libxl caller may return to the
application, with a child process still forked but not reaped libxl
contrary to the documented behaviour of libxl.
Instead, replace libxl__spawn_detach with libxl__spawn_initiate_detach
which is asynchronous. The detachable spawned children are abolished;
instead, we defer calling back to the in-libxl user until the middle
child has been reaped.
Also, remove erroneous comment suggesting that `death'' callback
parameter to libxl__ev_child_fork may be NULL. It may not, and there
are no callers which pass NULL.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Changes in v4:
* Clarify semantics of sub-states of Attached.
* Fix reference to `Failing'' sub-state in comment to
`Failed''.
Changes in v3 of series:
* Now also remove erroneous comment about NULL fork death callback.
---
tools/libxl/libxl_dm.c | 14 ++++-
tools/libxl/libxl_exec.c | 130 +++++++++++++++++++++++++-----------------
tools/libxl/libxl_internal.h | 43 +++++++-------
3 files changed, 110 insertions(+), 77 deletions(-)
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 340fcfa..b3de145 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -908,6 +908,8 @@ static void device_model_confirm(libxl__egc *egc,
libxl__spawn_state *spawn,
const char *xsdata);
static void device_model_startup_failed(libxl__egc *egc,
libxl__spawn_state *spawn);
+static void device_model_detached(libxl__egc *egc,
+ libxl__spawn_state *spawn);
/* our "next step" function, called from those callbacks and
elsewhere */
static void device_model_spawn_outcome(libxl__egc *egc,
@@ -1015,6 +1017,7 @@ retry_transaction:
spawn->midproc_cb = libxl__spawn_record_pid;
spawn->confirm_cb = device_model_confirm;
spawn->failure_cb = device_model_startup_failed;
+ spawn->detached_cb = device_model_detached;
rc = libxl__spawn_spawn(egc, spawn);
if (rc < 0)
@@ -1048,9 +1051,7 @@ static void device_model_confirm(libxl__egc *egc,
libxl__spawn_state *spawn,
if (strcmp(xsdata, "running"))
return;
- libxl__spawn_detach(gc, spawn);
-
- device_model_spawn_outcome(egc, dmss, 0);
+ libxl__spawn_initiate_detach(gc, spawn);
}
static void device_model_startup_failed(libxl__egc *egc,
@@ -1060,6 +1061,13 @@ static void device_model_startup_failed(libxl__egc *egc,
device_model_spawn_outcome(egc, dmss, ERROR_FAIL);
}
+static void device_model_detached(libxl__egc *egc,
+ libxl__spawn_state *spawn)
+{
+ libxl__dm_spawn_state *dmss = CONTAINER_OF(spawn, *dmss, spawn);
+ device_model_spawn_outcome(egc, dmss, 0);
+}
+
static void device_model_spawn_outcome(libxl__egc *egc,
libxl__dm_spawn_state *dmss,
int rc)
diff --git a/tools/libxl/libxl_exec.c b/tools/libxl/libxl_exec.c
index cfa379c..0477386 100644
--- a/tools/libxl/libxl_exec.c
+++ b/tools/libxl/libxl_exec.c
@@ -238,15 +238,22 @@ err:
/*
* Full set of possible states of a libxl__spawn_state and its _detachable:
*
- * ss-> ss-> ss-> | ssd->
ssd->
- * timeout xswatch ssd | mid ss
- * - Undefined undef undef no | - -
- * - Idle Idle Idle no | - -
- * - Active Active Active yes | Active yes
- * - Partial Active/Idle Active/Idle maybe | Active/Idle yes (if
exists)
- * - Detached - - - | Active no
+ * detaching failed mid timeout xswatch
+ * - Undefined undef undef - undef undef
+ * - Idle any any Idle Idle Idle
+ * - Attached OK 0 0 Active Active Active
+ * - Attached Failed 0 1 Active Idle Idle
+ * - Detaching 1 maybe Active Idle Idle
+ * - Partial any any Idle Active/Idle Active/Idle
*
- * When in state Detached, the middle process has been sent a SIGKILL.
+ * When in states Detaching or Attached Failed, the middle process has
+ * been sent a SIGKILL.
+ *
+ * The difference between Attached OK and Attached Failed is not
+ * directly visible to callers - callers see these two the same,
+ * although of course Attached OK will hopefully eventually result in
+ * a call to detached_cb, whereas Attached Failed will end up
+ * in a call to failure_cb.
*/
/* Event callbacks. */
@@ -257,19 +264,18 @@ static void spawn_timeout(libxl__egc *egc, libxl__ev_time
*ev,
static void spawn_middle_death(libxl__egc *egc, libxl__ev_child *childw,
pid_t pid, int status);
-/* Precondition: Partial. Results: Detached. */
+/* Precondition: Partial. Results: Idle. */
static void spawn_cleanup(libxl__gc *gc, libxl__spawn_state *ss);
-/* Precondition: Partial; caller has logged failure reason.
- * Results: Caller notified of failure;
- * after return, ss may be completely invalid as caller may reuse it */
-static void spawn_failed(libxl__egc *egc, libxl__spawn_state *ss);
+/* Precondition: Attached or Detaching; caller has logged failure reason.
+ * Results: Detaching, or Attached Failed */
+static void spawn_fail(libxl__egc *egc, libxl__spawn_state *ss);
void libxl__spawn_init(libxl__spawn_state *ss)
{
+ libxl__ev_child_init(&ss->mid);
libxl__ev_time_init(&ss->timeout);
libxl__ev_xswatch_init(&ss->xswatch);
- ss->ssd = 0;
}
int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *ss)
@@ -280,8 +286,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
int status, rc;
libxl__spawn_init(ss);
- ss->ssd = libxl__zalloc(NOGC, sizeof(*ss->ssd));
- libxl__ev_child_init(&ss->ssd->mid);
+ ss->failed = ss->detaching = 0;
rc = libxl__ev_time_register_rel(gc, &ss->timeout,
spawn_timeout, ss->timeout_ms);
@@ -291,7 +296,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
spawn_watch_event, ss->xspath);
if (rc) goto out_err;
- pid_t middle = libxl__ev_child_fork(gc, &ss->ssd->mid,
spawn_middle_death);
+ pid_t middle = libxl__ev_child_fork(gc, &ss->mid,
spawn_middle_death);
if (middle ==-1) { rc = ERROR_FAIL; goto out_err; }
if (middle) {
@@ -344,54 +349,64 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
static void spawn_cleanup(libxl__gc *gc, libxl__spawn_state *ss)
{
+ assert(!libxl__ev_child_inuse(&ss->mid));
+ libxl__ev_time_deregister(gc, &ss->timeout);
+ libxl__ev_xswatch_deregister(gc, &ss->xswatch);
+}
+
+static void spawn_detach(libxl__gc *gc, libxl__spawn_state *ss)
+/* Precondition: Attached or Detaching, but caller must have just set
+ * at least one of detaching or failed.
+ * Results: Detaching or Attached Failed */
+{
int r;
+ assert(libxl__ev_child_inuse(&ss->mid));
libxl__ev_time_deregister(gc, &ss->timeout);
libxl__ev_xswatch_deregister(gc, &ss->xswatch);
- libxl__spawn_state_detachable *ssd = ss->ssd;
- if (ssd) {
- if (libxl__ev_child_inuse(&ssd->mid)) {
- pid_t child = ssd->mid.pid;
- r = kill(child, SIGKILL);
- if (r && errno != ESRCH)
- LOGE(WARN, "%s: failed to kill intermediate child
(pid=%lu)",
- ss->what, (unsigned long)child);
- }
+ pid_t child = ss->mid.pid;
+ r = kill(child, SIGKILL);
+ if (r && errno != ESRCH)
+ LOGE(WARN, "%s: failed to kill intermediate child (pid=%lu)",
+ ss->what, (unsigned long)child);
+}
- /* disconnect the ss and ssd from each other */
- ssd->ss = 0;
- ss->ssd = 0;
- }
+void libxl__spawn_initiate_detach(libxl__gc *gc, libxl__spawn_state *ss)
+{
+ ss->detaching = 1;
+ spawn_detach(gc, ss);
}
-static void spawn_failed(libxl__egc *egc, libxl__spawn_state *ss)
+static void spawn_fail(libxl__egc *egc, libxl__spawn_state *ss)
+/* Caller must have logged. Must be last thing in calling function,
+ * as it may make the callback. Precondition: Attached or Detaching. */
{
EGC_GC;
- spawn_cleanup(gc, ss);
- ss->failure_cb(egc, ss); /* must be last; callback may do anything to ss
*/
+ ss->failed = 1;
+ spawn_detach(gc, ss);
}
static void spawn_timeout(libxl__egc *egc, libxl__ev_time *ev,
const struct timeval *requested_abs)
{
- /* Before event, was Active; is now Partial. */
+ /* Before event, was Attached. */
EGC_GC;
libxl__spawn_state *ss = CONTAINER_OF(ev, *ss, timeout);
LOG(ERROR, "%s: startup timed out", ss->what);
- spawn_failed(egc, ss); /* must be last */
+ spawn_fail(egc, ss); /* must be last */
}
static void spawn_watch_event(libxl__egc *egc, libxl__ev_xswatch *xsw,
const char *watch_path, const char *event_path)
{
- /* On entry, is Active. */
+ /* On entry, is Attached. */
EGC_GC;
libxl__spawn_state *ss = CONTAINER_OF(xsw, *ss, xswatch);
char *p = libxl__xs_read(gc, 0, ss->xspath);
if (!p && errno != ENOENT) {
LOG(ERROR, "%s: xenstore read of %s failed", ss->what,
ss->xspath);
- spawn_failed(egc, ss); /* must be last */
+ spawn_fail(egc, ss); /* must be last */
return;
}
ss->confirm_cb(egc, ss, p); /* must be last */
@@ -399,20 +414,22 @@ static void spawn_watch_event(libxl__egc *egc,
libxl__ev_xswatch *xsw,
static void spawn_middle_death(libxl__egc *egc, libxl__ev_child *childw,
pid_t pid, int status)
- /* Before event, was Active or Detached;
- * is now Active or Detached except that ssd->mid is Idle */
+ /* On entry, is Attached or Detaching */
{
EGC_GC;
- libxl__spawn_state_detachable *ssd = CONTAINER_OF(childw, *ssd, mid);
- libxl__spawn_state *ss = ssd->ss;
-
- if (!WIFEXITED(status)) {
+ libxl__spawn_state *ss = CONTAINER_OF(childw, *ss, mid);
+
+ if ((ss->failed || ss->detaching) &&
+ ((WIFEXITED(status) && WEXITSTATUS(status)==0) ||
+ (WIFSIGNALED(status) && WTERMSIG(status)==SIGKILL))) {
+ /* as expected */
+ } else if (!WIFEXITED(status)) {
+ int loglevel = ss->detaching ? XTL_WARN : XTL_ERROR;
const char *what - GCSPRINTF("%s intermediate process
(startup monitor)",
- ss ? ss->what : "(detached)");
- int loglevel = ss ? XTL_ERROR : XTL_WARN;
+ GCSPRINTF("%s intermediate process (startup monitor)",
ss->what);
libxl_report_child_exitstatus(CTX, loglevel, what, pid, status);
- } else if (ss) { /* otherwise it was supposed to be a daemon by now */
+ ss->failed = 1;
+ } else {
if (!status)
LOG(ERROR, "%s [%ld]: unexpectedly exited with exit status
0,"
" when we were waiting for it to confirm startup",
@@ -430,15 +447,22 @@ static void spawn_middle_death(libxl__egc *egc,
libxl__ev_child *childw,
LOG(ERROR, "%s [%ld]: died during startup due to unknown
fatal"
" signal number %d", ss->what, (unsigned
long)pid, sig);
}
- ss->ssd = 0; /* detatch the ssd to make the ss be in state Partial
*/
- spawn_failed(egc, ss); /* must be last use of ss */
+ ss->failed = 1;
}
- free(ssd);
-}
-void libxl__spawn_detach(libxl__gc *gc, libxl__spawn_state *ss)
-{
spawn_cleanup(gc, ss);
+
+ if (ss->failed && !ss->detaching) {
+ ss->failure_cb(egc, ss); /* must be last */
+ return;
+ }
+
+ if (ss->failed && ss->detaching)
+ LOG(WARN,"%s underlying machinery seemed to fail,"
+ " but its function seems to have been successful",
ss->what);
+
+ assert(ss->detaching);
+ ss->detached_cb(egc, ss);
}
/*
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 85f4bc6..9df0db5 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -690,8 +690,7 @@ static inline int libxl__ev_xswatch_isregistered(const
libxl__ev_xswatch *xw)
* the libxl event machinery.
*
* The parent may signal the child but it must not reap it. That will
- * be done by the event machinery. death may be NULL in which case
- * the child is still reaped but its death is ignored.
+ * be done by the event machinery.
*
* It is not possible to "deregister" the child death event source.
* It will generate exactly one event callback; until then the childw
@@ -998,8 +997,8 @@ _hidden int libxl__device_pci_destroy_all(libxl__gc *gc,
uint32_t domid);
*
* Higher-level double-fork and separate detach eg as for device models
*
- * Each libxl__spawn_state is in one of the conventional states
- * Undefined, Idle, Active
+ * Each libxl__spawn_state is in one of these states
+ * Undefined, Idle, Attached, Detaching
*/
typedef struct libxl__obsolete_spawn_starting libxl__spawn_starting;
@@ -1040,15 +1039,15 @@ _hidden void libxl__spawn_init(libxl__spawn_state*);
* intermediate or final child; an error message will have been
* logged.
*
- * confirm_cb and failure_cb will not be called reentrantly from
- * within libxl__spawn_spawn.
+ * confirm_cb, failure_cb and detached_cb will not be called
+ * reentrantly from within libxl__spawn_spawn.
*
* what: string describing the spawned process, used for logging
*
* Logs errors. A copy of "what" is taken.
* Return values:
* < 0 error, *spawn is now Idle and need not be detached
- * +1 caller is the parent, *spawn is Active and must eventually be
detached
+ * +1 caller is the parent, *spawn is Attached and must be detached
* 0 caller is now the inner child, should probably call libxl__exec
*
* The spawn state must be Undefined or Idle on entry.
@@ -1056,12 +1055,15 @@ _hidden void libxl__spawn_init(libxl__spawn_state*);
_hidden int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *spawn);
/*
- * libxl__spawn_detach - Detaches the daemonic child.
+ * libxl__spawn_request_detach - Detaches the daemonic child.
*
* Works by killing the intermediate process from spawn_spawn.
* After this function returns, failures of either child are no
* longer reported via failure_cb.
*
+ * This is not synchronous: there will be a further callback when
+ * the detach is complete.
+ *
* If called before the inner child has been created, this may prevent
* it from running at all. Thus this should be called only when the
* inner child has notified that it is ready. Normally it will be
@@ -1069,10 +1071,10 @@ _hidden int libxl__spawn_spawn(libxl__egc *egc,
libxl__spawn_state *spawn);
*
* Logs errors.
*
- * The spawn state must be Active or Idle on entry and will be Idle
+ * The spawn state must be Attached entry and will be Detaching
* on return.
*/
-_hidden void libxl__spawn_detach(libxl__gc *gc, libxl__spawn_state*);
+_hidden void libxl__spawn_initiate_detach(libxl__gc *gc, libxl__spawn_state*);
/*
* If successful, this should return 0.
@@ -1109,15 +1111,11 @@ typedef void libxl__spawn_failure_cb(libxl__egc*,
libxl__spawn_state*);
typedef void libxl__spawn_confirm_cb(libxl__egc*, libxl__spawn_state*,
const char *xsdata);
-typedef struct {
- /* Private to the spawn implementation.
- */
- /* This separate struct, from malloc, allows us to "detach"
- * the child and reap it later, when our user has gone
- * away and freed its libxl__spawn_state */
- struct libxl__spawn_state *ss;
- libxl__ev_child mid;
-} libxl__spawn_state_detachable;
+/*
+ * Called when the detach (requested by libxl__spawn_initiate_detach) has
+ * completed. On entry to the callback the spawn state is Idle.
+ */
+typedef void libxl__spawn_detached_cb(libxl__egc*, libxl__spawn_state*);
struct libxl__spawn_state {
/* must be filled in by user and remain valid */
@@ -1129,15 +1127,18 @@ struct libxl__spawn_state {
libxl__spawn_midproc_cb *midproc_cb;
libxl__spawn_failure_cb *failure_cb;
libxl__spawn_confirm_cb *confirm_cb;
+ libxl__spawn_detached_cb *detached_cb;
/* remaining fields are private to libxl_spawn_... */
+ int detaching; /* we are in Detaching */
+ int failed; /* might be true whenever we are not Idle */
+ libxl__ev_child mid; /* always in use whenever we are not Idle */
libxl__ev_time timeout;
libxl__ev_xswatch xswatch;
- libxl__spawn_state_detachable *ssd;
};
static inline int libxl__spawn_inuse(libxl__spawn_state *ss)
- { return !!ss->ssd; }
+ { return libxl__ev_child_inuse(&ss->mid); }
/*
* libxl_spawner_record_pid - Record given pid in xenstore
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:54 UTC
[PATCH 19/21] libxl: do not leak an event struct on ignored ao progress
On entry to libxl__ao_progress_report, the caller has allocated an
event. If the progress report is to be ignored, we need to free it.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl_event.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index eb23a93..1957505 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -1602,6 +1602,7 @@ void libxl__ao_progress_report(libxl__egc *egc, libxl__ao
*ao,
ev->for_user = how->for_event;
if (how->callback == dummy_asyncprogress_callback_ignore) {
LOG(DEBUG,"ao %p: progress report: ignored",ao);
+ libxl_event_free(CTX,ev);
/* ignore */
} else if (how->callback) {
libxl__aop_occurred *aop = libxl__zalloc(&egc->gc,
sizeof(*aop));
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:54 UTC
[PATCH 20/21] libxl: further fixups re LIBXL_DOMAIN_TYPE
* Abolish the macro LIBXL__DOMAIN_IS_TYPE which had incorrect error
handlng. At every call site, replace it with an open-coded call to
libxl_domain_type and check against LIBXL_DOMAIN_TYPE_INVALID.
* This involves adding an `out:'' to libxl_domain_unpause.
* In libxl_domain_destroy and do_pci_add, do not `default: abort();''
if the domain type cannot be found. Instead switch on
LIBXL_DOMAIN_TYPE_INVALID specifically and do some actual error
handling.
* In libxl__primary_console_find, remove a spurious default clause
from the domain type switch.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v4 of series:
* Hunk
In libxl_domain_suspend (as reorganised) error check, check for
LIBXL_DOMAIN_TYPE_INVALID and remove a pointless extra log message.
merged into the earlier patch where the slightly-wrong code was
introduced.
---
tools/libxl/libxl.c | 30 +++++++++++++++++++++++-------
tools/libxl/libxl_internal.h | 5 +++--
tools/libxl/libxl_pci.c | 18 +++++++++++++-----
3 files changed, 39 insertions(+), 14 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index d3b017b..cd2dbda 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -390,7 +390,13 @@ int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int
suspend_cancel)
goto out;
}
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, HVM)) {
+ libxl_domain_type type = libxl__domain_type(gc, domid);
+ if (type == LIBXL_DOMAIN_TYPE_INVALID) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (type == LIBXL_DOMAIN_TYPE_HVM) {
rc = libxl__domain_resume_device_model(gc, domid);
if (rc) {
LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
@@ -788,7 +794,13 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid)
char *state;
int ret, rc = 0;
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, HVM)) {
+ libxl_domain_type type = libxl__domain_type(gc, domid);
+ if (type == LIBXL_DOMAIN_TYPE_INVALID) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (type == LIBXL_DOMAIN_TYPE_HVM) {
path = libxl__sprintf(gc,
"/local/domain/0/device-model/%d/state", domid);
state = libxl__xs_read(gc, XBT_NULL, path);
if (state != NULL && !strcmp(state, "paused")) {
@@ -802,6 +814,7 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid)
LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unpausing domain
%d", domid);
rc = ERROR_FAIL;
}
+ out:
GC_FREE;
return rc;
}
@@ -813,7 +826,11 @@ int libxl__domain_pvcontrol_available(libxl__gc *gc,
uint32_t domid)
unsigned long pvdriver = 0;
int ret;
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV))
+ libxl_domain_type domtype = libxl__domain_type(gc, domid);
+ if (domtype == LIBXL_DOMAIN_TYPE_INVALID)
+ return ERROR_FAIL;
+
+ if (domtype == LIBXL_DOMAIN_TYPE_PV)
return 1;
ret = xc_get_hvm_param(ctx->xch, domid, HVM_PARAM_CALLBACK_IRQ,
&pvdriver);
@@ -1213,8 +1230,9 @@ int libxl_domain_destroy(libxl_ctx *ctx, uint32_t domid)
pid = libxl__xs_read(gc, XBT_NULL, libxl__sprintf(gc,
"/local/domain/%d/image/device-model-pid", domid));
dm_present = (pid != NULL);
break;
- default:
- abort();
+ case LIBXL_DOMAIN_TYPE_INVALID:
+ rc = ERROR_FAIL;
+ goto out;
}
dom_path = libxl__xs_get_dompath(gc, domid);
@@ -1362,8 +1380,6 @@ static int libxl__primary_console_find(libxl_ctx *ctx,
uint32_t domid_vm,
case LIBXL_DOMAIN_TYPE_INVALID:
rc = ERROR_INVAL;
goto out;
- default:
- abort();
}
}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9df0db5..36c75ed 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -797,8 +797,7 @@ _hidden int libxl__domain_cpupool(libxl__gc *gc, uint32_t
domid);
_hidden libxl_scheduler libxl__domain_scheduler(libxl__gc *gc, uint32_t domid);
_hidden int libxl__sched_set_params(libxl__gc *gc, uint32_t domid,
libxl_domain_sched_params *scparams);
-#define LIBXL__DOMAIN_IS_TYPE(gc, domid, type) \
- libxl__domain_type((gc), (domid)) == LIBXL_DOMAIN_TYPE_##type
+
typedef struct {
uint32_t store_port;
uint32_t store_domid;
@@ -841,7 +840,9 @@ _hidden int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid);
_hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid);
+/* returns 0 or 1, or a libxl error code */
_hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid);
+
_hidden char * libxl__domain_pvcontrol_read(libxl__gc *gc,
xs_transaction_t t, uint32_t
domid);
_hidden int libxl__domain_pvcontrol_write(libxl__gc *gc, xs_transaction_t t,
diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c
index de1b79f..81438be 100644
--- a/tools/libxl/libxl_pci.c
+++ b/tools/libxl/libxl_pci.c
@@ -128,7 +128,11 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc,
uint32_t domid, libxl_d
if (!num_devs)
return libxl__create_pci_backend(gc, domid, pcidev, 1);
- if (!starting && LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) {
+ libxl_domain_type domtype = libxl__domain_type(gc, domid);
+ if (domtype == LIBXL_DOMAIN_TYPE_INVALID)
+ return ERROR_FAIL;
+
+ if (!starting && domtype == LIBXL_DOMAIN_TYPE_PV) {
if (libxl__wait_for_backend(gc, be_path, "4") < 0)
return ERROR_FAIL;
}
@@ -171,7 +175,11 @@ static int libxl__device_pci_remove_xenstore(libxl__gc *gc,
uint32_t domid, libx
return ERROR_INVAL;
num = atoi(num_devs);
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) {
+ libxl_domain_type domtype = libxl__domain_type(gc, domid);
+ if (domtype == LIBXL_DOMAIN_TYPE_INVALID)
+ return ERROR_FAIL;
+
+ if (domtype == LIBXL_DOMAIN_TYPE_PV) {
if (libxl__wait_for_backend(gc, be_path, "4") < 0) {
LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not
ready", be_path);
return ERROR_FAIL;
@@ -199,7 +207,7 @@ retry_transaction:
if (errno == EAGAIN)
goto retry_transaction;
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) {
+ if (domtype == LIBXL_DOMAIN_TYPE_PV) {
if (libxl__wait_for_backend(gc, be_path, "4") < 0) {
LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not
ready", be_path);
return ERROR_FAIL;
@@ -939,8 +947,8 @@ static int do_pci_add(libxl__gc *gc, uint32_t domid,
libxl_device_pci *pcidev, i
}
break;
}
- default:
- abort();
+ case LIBXL_DOMAIN_TYPE_INVALID:
+ return ERROR_FAIL;
}
out:
if (!libxl_is_stubdom(ctx, domid, NULL)) {
--
1.7.2.5
Ian Jackson
2012-Jun-15 11:54 UTC
[PATCH 21/21] libxl: DO NOT APPLY enforce prohibition on internal
DO NOT APPLY THIS PATCH.
It contains -Wno-error. Without that it would break the build.
Subject [PATCH] libxl: enforce prohibitions of internal callers
libxl_internal.h says:
* Functions using LIBXL__INIT_EGC may *not* generally be called from
* within libxl, because libxl__egc_cleanup may call back into the
* application. ...
and
* ... [Functions which take an ao_how] MAY NOT
* be called from inside libxl, because they can cause reentrancy
* callbacks.
However, this was not enforced. Particularly the latter restriction
is easy to overlook, especially since during the transition period to
the new event system we have bent this rule a couple of times, and the
bad pattern simply involves passing 0 or NULL for the ao_how.
So use the compiler to enforce this property, as follows:
- Mark all functions which take a libxl_asyncop_how, or which
use EGC_INIT or LIBXL__INIT_EGC, with a new annotation
LIBXL_EXTERNAL_CALLERS_ONLY in the public header.
- Change the documentation comment for asynch operations and egcs to
say that this should always be done.
- Arrange that if libxl.h is included via libxl_internal.h,
LIBXL_EXTERNAL_CALLERS_ONLY expands to __attribute__((warning(...))),
which generates a message like this:
libxl.c:1772: warning: call to
''libxl_device_disk_remove''
declared with attribute warning:
may not be called from within libxl
Otherwise, the annotation expands to nothing, so external
callers are unaffected.
- Forbid inclusion of both libxl.h and libxl_internal.h unless
libxl_internal.h came first, so that the above check doesn''t have
any loopholes. Files which include libxl_internal.h should not
include libxl.h as well.
This is enforced explicitly using #error. However, in practice
with the current tree it just changes the error message when this
mistake is made; otherwise we would carry on to immediately
following #define which would cause the compiler to complain that
LIBXL_EXTERNAL_CALLERS_ONLY was redefined. Then the developer
might be tempted to add a #ifndef which would be wrong - it would
leave the affected translation unit unprotected by the new
enforcement regime. So let''s be explicit.
- Fix the one source of files which violate the above principle, the
output from the idl compiler, by removing the redundant inclusion
of libxl.h from the output.
In the tree I am using as a base at the time of writing, this new
restriction catches three errors: two in libxl_device_disk_remove and
one in libxl__device_disk_local_detach. To avoid entirely breaking my
build I have also done this:
- Temporarily change -Werror to -Wno-error in the libxl Makefile.
This patch should not be applied in this form.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/Makefile | 2 +-
tools/libxl/gentypes.py | 1 -
tools/libxl/libxl.h | 34 +++++++++++++++++++++++++---------
tools/libxl/libxl_event.h | 21 ++++++++++++++-------
tools/libxl/libxl_internal.h | 14 ++++++++++----
5 files changed, 50 insertions(+), 22 deletions(-)
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index ddc2624..c238ac0 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -11,7 +11,7 @@ MINOR = 0
XLUMAJOR = 1.0
XLUMINOR = 0
-CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \
+CFLAGS += -Wno-error -Wno-format-zero-length -Wmissing-declarations \
-Wno-declaration-after-statement -Wformat-nonliteral
CFLAGS += -I. -fPIC
diff --git a/tools/libxl/gentypes.py b/tools/libxl/gentypes.py
index 3c561ba..6e83b21 100644
--- a/tools/libxl/gentypes.py
+++ b/tools/libxl/gentypes.py
@@ -341,7 +341,6 @@ if __name__ == ''__main__'':
#include <stdlib.h>
#include <string.h>
-#include "libxl.h"
#include "libxl_internal.h"
#define LIBXL_DTOR_POISON 0xa5
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 10d7115..1a32d9e 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -266,6 +266,13 @@
#endif
#endif
+/* Functions annotated with LIBXL_EXTERNAL_CALLERS_ONLY may not be
+ * called from within libxl itself. Callers outside libxl, who
+ * do not #include libxl_internal.h, are fine. */
+#ifndef LIBXL_EXTERNAL_CALLERS_ONLY
+#define LIBXL_EXTERNAL_CALLERS_ONLY /* disappears for callers outside libxl */
+#endif
+
typedef uint8_t libxl_mac[6];
#define LIBXL_MAC_FMT "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx"
#define LIBXL_MAC_FMTLEN ((2*6)+5) /* 6 hex bytes plus 5 colons */
@@ -495,11 +502,13 @@ int libxl_ctx_free(libxl_ctx *ctx /* 0 is OK */);
int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid,
const libxl_asyncop_how *ao_how,
- const libxl_asyncprogress_how *aop_console_how);
+ const libxl_asyncprogress_how *aop_console_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd,
const libxl_asyncop_how *ao_how,
- const libxl_asyncprogress_how
*aop_console_how);
+ const libxl_asyncprogress_how *aop_console_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* A progress report will be made via ao_console_how, of type
* domain_create_console_available, when the domain''s primary
* console is available and can be connected to.
@@ -510,7 +519,8 @@ void libxl_domain_config_dispose(libxl_domain_config
*d_config);
int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd,
int flags, /* LIBXL_SUSPEND_* */
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
#define LIBXL_SUSPEND_DEBUG 1
#define LIBXL_SUSPEND_LIVE 2
@@ -522,7 +532,8 @@ int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int
suspend_cancel);
int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
uint32_t domid, int send_fd, int recv_fd,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_shutdown(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_reboot(libxl_ctx *ctx, uint32_t domid);
@@ -544,7 +555,8 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_core_dump(libxl_ctx *ctx, uint32_t domid,
const char *filename,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint32_t
target_memkb);
int libxl_set_memory_target(libxl_ctx *ctx, uint32_t domid, int32_t
target_memkb, int relative, int enforce);
@@ -653,7 +665,8 @@ void libxl_vminfo_list_free(libxl_vminfo *list, int nr);
int libxl_device_disk_add(libxl_ctx *ctx, uint32_t domid, libxl_device_disk
*disk);
int libxl_device_disk_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_disk_destroy(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk);
@@ -671,7 +684,8 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk);
int libxl_device_nic_add(libxl_ctx *ctx, uint32_t domid, libxl_device_nic
*nic);
int libxl_device_nic_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_nic *nic,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_nic_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_nic
*nic);
libxl_device_nic *libxl_device_nic_list(libxl_ctx *ctx, uint32_t domid, int
*num);
@@ -682,14 +696,16 @@ int libxl_device_nic_getinfo(libxl_ctx *ctx, uint32_t
domid,
int libxl_device_vkb_add(libxl_ctx *ctx, uint32_t domid, libxl_device_vkb
*vkb);
int libxl_device_vkb_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_vkb *vkb,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_vkb_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_vkb
*vkb);
/* Framebuffer */
int libxl_device_vfb_add(libxl_ctx *ctx, uint32_t domid, libxl_device_vfb
*vfb);
int libxl_device_vfb_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_vfb *vfb,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_vfb_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_vfb
*vfb);
/* PCI Passthrough */
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index 713d96d..3344bc8 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -37,7 +37,8 @@ typedef int libxl_event_predicate(const libxl_event*, void
*user);
int libxl_event_check(libxl_ctx *ctx, libxl_event **event_r,
uint64_t typemask,
- libxl_event_predicate *predicate, void *predicate_user);
+ libxl_event_predicate *predicate, void *predicate_user)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* Searches for an event, already-happened, which matches typemask
* and predicate. predicate==0 matches any event.
* libxl_event_check returns the event, which must then later be
@@ -48,7 +49,8 @@ int libxl_event_check(libxl_ctx *ctx, libxl_event **event_r,
int libxl_event_wait(libxl_ctx *ctx, libxl_event **event_r,
uint64_t typemask,
- libxl_event_predicate *predicate, void *predicate_user);
+ libxl_event_predicate *predicate, void *predicate_user)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* Like libxl_event_check but blocks if no suitable events are
* available, until some are. Uses libxl_osevent_beforepoll/
* _afterpoll so may be inefficient if very many domains are being
@@ -256,7 +258,8 @@ struct pollfd;
*/
int libxl_osevent_beforepoll(libxl_ctx *ctx, int *nfds_io,
struct pollfd *fds, int *timeout_upd,
- struct timeval now);
+ struct timeval now)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* nfds and fds[0..nfds] must be from the most recent call to
* _beforepoll, as modified by poll. (It is therefore not possible
@@ -271,7 +274,8 @@ int libxl_osevent_beforepoll(libxl_ctx *ctx, int *nfds_io,
* libxl_event_check.
*/
void libxl_osevent_afterpoll(libxl_ctx *ctx, int nfds, const struct pollfd
*fds,
- struct timeval now);
+ struct timeval now)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
typedef struct libxl_osevent_hooks {
@@ -357,14 +361,16 @@ void libxl_osevent_register_hooks(libxl_ctx *ctx,
*/
void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
- int fd, short events, short revents);
+ int fd, short events, short revents)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* Implicitly, on entry to this function the timeout has been
* deregistered. If _occurred_timeout is called, libxl will not
* call timeout_deregister; if it wants to requeue the timeout it
* will call timeout_register again.
*/
-void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
+void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/*======================================================================*/
@@ -506,7 +512,8 @@ void libxl_childproc_setmode(libxl_ctx *ctx, const
libxl_childproc_hooks *hooks,
* certainly need to use the self-pipe trick (or a working pselect or
* ppoll) to implement this.
*/
-int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status);
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/*
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 36c75ed..6c859bc 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -52,6 +52,12 @@
#include <xen/io/xenbus.h>
+#ifdef LIBXL_H
+# error libxl.h should be included via libxl_internal.h, not separately
+#endif
+#define LIBXL_EXTERNAL_CALLERS_ONLY \
+ __attribute__((warning("may not be called from within libxl")))
+
#include "libxl.h"
#include "_paths.h"
#include "_libxl_save_msgs_callout.h"
@@ -1538,10 +1544,10 @@ libxl__device_model_version_running(libxl__gc *gc,
uint32_t domid);
*
* Functions using LIBXL__INIT_EGC may *not* generally be called from
* within libxl, because libxl__egc_cleanup may call back into the
- * application. This should be documented near the function
- * prototype(s) for callers of LIBXL__INIT_EGC and EGC_INIT. You
- * should in any case not find it necessary to call egc-creators from
- * within libxl.
+ * application. This should be enforced by declaring all such
+ * functions in libxl.h or libxl_event.h with
+ * LIBXL_EXTERNAL_CALLERS_ONLY. You should in any case not find it
+ * necessary to call egc-creators from within libxl.
*
* The callbacks must all take place with the ctx unlocked because
* the application is entitled to reenter libxl from them. This
--
1.7.2.5
Stefano Stabellini
2012-Jun-15 13:40 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Fri, 15 Jun 2012, Ian Jackson wrote:> This is v4 of my series to asyncify save/restore. All comments have > been addressed. > > I have retested this with HVM (RHEL) and PV guests, for both localhost > migration and save/restore. RHEL seems to have some unreliability > which occurs with the baseline too. I was not able to test stub dms > at all because when trying to boot my RHEL guest with a stub dm the dm > domain immediately crashed.IanC, didn''t you test xen-unstable with stubdoms not too long ago? In any case we should probably add "working stubdoms" to the blocker list.
Ian Campbell
2012-Jun-21 08:47 UTC
Re: [PATCH 02/21] libxc: Do not segfault if (e.g.) switch_qemu_logdirty fails
On Fri, 2012-06-15 at 12:53 +0100, Ian Jackson wrote:> In xc_domain_save the local variable `ob'' is initialised to NULL. > There are then various startup actions. Some of these `goto out'' on > failure; for example the call to callbacks->switch_qemu_logdirty on > l.978. However, out is used both by success and error paths. So it > attempts (l.2043) to flush the current output buffer. If ob has not > yet been assigned a non-NULL value, this segfaults. So make the call > to outbuf_flush conditional on ob. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Doing anything like splitting the error case into a separate path etc looks too complex to me, so this seems like the best change. Acked-by: Ian Campbell <ian.campbell@citrix.com> I couldn''t help noticing that there are a bunch of "goto out" statements _after_ the out label. They all appear to be protected by an if (!rc) and be preceded by an rc = 1, but talk about making it hard for the reader!> --- > tools/libxc/xc_domain_save.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/tools/libxc/xc_domain_save.c b/tools/libxc/xc_domain_save.c > index fcc7718..c359649 100644 > --- a/tools/libxc/xc_domain_save.c > +++ b/tools/libxc/xc_domain_save.c > @@ -2040,7 +2040,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter > } > > /* Flush last write and discard cache for file. */ > - if ( outbuf_flush(xch, ob, io_fd) < 0 ) { > + if ( ob && outbuf_flush(xch, ob, io_fd) < 0 ) { > PERROR("Error when flushing output buffer"); > rc = 1; > }
Ian Campbell
2012-Jun-22 11:42 UTC
Re: [PATCH 20/21] libxl: further fixups re LIBXL_DOMAIN_TYPE
On Fri, 2012-06-15 at 12:54 +0100, Ian Jackson wrote:> * Abolish the macro LIBXL__DOMAIN_IS_TYPE which had incorrect error > handlng. At every call site, replace it with an open-coded call tohandling> libxl_domain_type and check against LIBXL_DOMAIN_TYPE_INVALID. > > * This involves adding an `out:'' to libxl_domain_unpause. > > * In libxl_domain_destroy and do_pci_add, do not `default: abort();'' > if the domain type cannot be found. Instead switch on > LIBXL_DOMAIN_TYPE_INVALID specifically and do some actual error > handling. > > * In libxl__primary_console_find, remove a spurious default clause > from the domain type switch. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com>> > Changes in v4 of series: > * Hunk > In libxl_domain_suspend (as reorganised) error check, check for > LIBXL_DOMAIN_TYPE_INVALID and remove a pointless extra log message. > merged into the earlier patch where the slightly-wrong code was > introduced. > --- > tools/libxl/libxl.c | 30 +++++++++++++++++++++++------- > tools/libxl/libxl_internal.h | 5 +++-- > tools/libxl/libxl_pci.c | 18 +++++++++++++----- > 3 files changed, 39 insertions(+), 14 deletions(-) > > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c > index d3b017b..cd2dbda 100644 > --- a/tools/libxl/libxl.c > +++ b/tools/libxl/libxl.c > @@ -390,7 +390,13 @@ int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int suspend_cancel) > goto out; > } > > - if (LIBXL__DOMAIN_IS_TYPE(gc, domid, HVM)) { > + libxl_domain_type type = libxl__domain_type(gc, domid); > + if (type == LIBXL_DOMAIN_TYPE_INVALID) { > + rc = ERROR_FAIL; > + goto out; > + } > + > + if (type == LIBXL_DOMAIN_TYPE_HVM) { > rc = libxl__domain_resume_device_model(gc, domid); > if (rc) { > LIBXL__LOG(ctx, LIBXL__LOG_ERROR, > @@ -788,7 +794,13 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid) > char *state; > int ret, rc = 0; > > - if (LIBXL__DOMAIN_IS_TYPE(gc, domid, HVM)) { > + libxl_domain_type type = libxl__domain_type(gc, domid); > + if (type == LIBXL_DOMAIN_TYPE_INVALID) { > + rc = ERROR_FAIL; > + goto out; > + } > + > + if (type == LIBXL_DOMAIN_TYPE_HVM) { > path = libxl__sprintf(gc, "/local/domain/0/device-model/%d/state", domid); > state = libxl__xs_read(gc, XBT_NULL, path); > if (state != NULL && !strcmp(state, "paused")) { > @@ -802,6 +814,7 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid) > LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unpausing domain %d", domid); > rc = ERROR_FAIL; > } > + out: > GC_FREE; > return rc; > } > @@ -813,7 +826,11 @@ int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid) > unsigned long pvdriver = 0; > int ret; > > - if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) > + libxl_domain_type domtype = libxl__domain_type(gc, domid); > + if (domtype == LIBXL_DOMAIN_TYPE_INVALID) > + return ERROR_FAIL; > + > + if (domtype == LIBXL_DOMAIN_TYPE_PV) > return 1; > > ret = xc_get_hvm_param(ctx->xch, domid, HVM_PARAM_CALLBACK_IRQ, &pvdriver); > @@ -1213,8 +1230,9 @@ int libxl_domain_destroy(libxl_ctx *ctx, uint32_t domid) > pid = libxl__xs_read(gc, XBT_NULL, libxl__sprintf(gc, "/local/domain/%d/image/device-model-pid", domid)); > dm_present = (pid != NULL); > break; > - default: > - abort(); > + case LIBXL_DOMAIN_TYPE_INVALID: > + rc = ERROR_FAIL; > + goto out; > } > > dom_path = libxl__xs_get_dompath(gc, domid); > @@ -1362,8 +1380,6 @@ static int libxl__primary_console_find(libxl_ctx *ctx, uint32_t domid_vm, > case LIBXL_DOMAIN_TYPE_INVALID: > rc = ERROR_INVAL; > goto out; > - default: > - abort(); > } > } > > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index 9df0db5..36c75ed 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -797,8 +797,7 @@ _hidden int libxl__domain_cpupool(libxl__gc *gc, uint32_t domid); > _hidden libxl_scheduler libxl__domain_scheduler(libxl__gc *gc, uint32_t domid); > _hidden int libxl__sched_set_params(libxl__gc *gc, uint32_t domid, > libxl_domain_sched_params *scparams); > -#define LIBXL__DOMAIN_IS_TYPE(gc, domid, type) \ > - libxl__domain_type((gc), (domid)) == LIBXL_DOMAIN_TYPE_##type > + > typedef struct { > uint32_t store_port; > uint32_t store_domid; > @@ -841,7 +840,9 @@ _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid); > > _hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid); > > +/* returns 0 or 1, or a libxl error code */ > _hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid); > + > _hidden char * libxl__domain_pvcontrol_read(libxl__gc *gc, > xs_transaction_t t, uint32_t domid); > _hidden int libxl__domain_pvcontrol_write(libxl__gc *gc, xs_transaction_t t, > diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c > index de1b79f..81438be 100644 > --- a/tools/libxl/libxl_pci.c > +++ b/tools/libxl/libxl_pci.c > @@ -128,7 +128,11 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d > if (!num_devs) > return libxl__create_pci_backend(gc, domid, pcidev, 1); > > - if (!starting && LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) { > + libxl_domain_type domtype = libxl__domain_type(gc, domid); > + if (domtype == LIBXL_DOMAIN_TYPE_INVALID) > + return ERROR_FAIL; > + > + if (!starting && domtype == LIBXL_DOMAIN_TYPE_PV) { > if (libxl__wait_for_backend(gc, be_path, "4") < 0) > return ERROR_FAIL; > } > @@ -171,7 +175,11 @@ static int libxl__device_pci_remove_xenstore(libxl__gc *gc, uint32_t domid, libx > return ERROR_INVAL; > num = atoi(num_devs); > > - if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) { > + libxl_domain_type domtype = libxl__domain_type(gc, domid); > + if (domtype == LIBXL_DOMAIN_TYPE_INVALID) > + return ERROR_FAIL; > + > + if (domtype == LIBXL_DOMAIN_TYPE_PV) { > if (libxl__wait_for_backend(gc, be_path, "4") < 0) { > LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); > return ERROR_FAIL; > @@ -199,7 +207,7 @@ retry_transaction: > if (errno == EAGAIN) > goto retry_transaction; > > - if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) { > + if (domtype == LIBXL_DOMAIN_TYPE_PV) { > if (libxl__wait_for_backend(gc, be_path, "4") < 0) { > LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); > return ERROR_FAIL; > @@ -939,8 +947,8 @@ static int do_pci_add(libxl__gc *gc, uint32_t domid, libxl_device_pci *pcidev, i > } > break; > } > - default: > - abort(); > + case LIBXL_DOMAIN_TYPE_INVALID: > + return ERROR_FAIL; > } > out: > if (!libxl_is_stubdom(ctx, domid, NULL)) {
Ian Campbell
2012-Jun-22 11:45 UTC
Re: [PATCH 21/21] libxl: DO NOT APPLY enforce prohibition on internal
On Fri, 2012-06-15 at 12:54 +0100, Ian Jackson wrote:> DO NOT APPLY THIS PATCH. > It contains -Wno-error. Without that it would break the build.I''m happy with the shape of this patch, and apart from the above caveat I''d be happy to Ack it, assuming it were preceded by fixes to those issues. I fear slightly that we will be continually forgetting to add the tag during review. Is there a way we could augment with run time checking? Perhaps LIBXL__INIT_EGC can tell if a CTX is not "outer" somehow?> > Subject [PATCH] libxl: enforce prohibitions of internal callers > > libxl_internal.h says: > > * Functions using LIBXL__INIT_EGC may *not* generally be called from > * within libxl, because libxl__egc_cleanup may call back into the > * application. ... > > and > > * ... [Functions which take an ao_how] MAY NOT > * be called from inside libxl, because they can cause reentrancy > * callbacks. > > However, this was not enforced. Particularly the latter restriction > is easy to overlook, especially since during the transition period to > the new event system we have bent this rule a couple of times, and the > bad pattern simply involves passing 0 or NULL for the ao_how. > > So use the compiler to enforce this property, as follows: > > - Mark all functions which take a libxl_asyncop_how, or which > use EGC_INIT or LIBXL__INIT_EGC, with a new annotation > LIBXL_EXTERNAL_CALLERS_ONLY in the public header. > > - Change the documentation comment for asynch operations and egcs to > say that this should always be done. > > - Arrange that if libxl.h is included via libxl_internal.h, > LIBXL_EXTERNAL_CALLERS_ONLY expands to __attribute__((warning(...))), > which generates a message like this: > libxl.c:1772: warning: call to ''libxl_device_disk_remove'' > declared with attribute warning: > may not be called from within libxl > Otherwise, the annotation expands to nothing, so external > callers are unaffected. > > - Forbid inclusion of both libxl.h and libxl_internal.h unless > libxl_internal.h came first, so that the above check doesn''t have > any loopholes. Files which include libxl_internal.h should not > include libxl.h as well. > > This is enforced explicitly using #error. However, in practice > with the current tree it just changes the error message when this > mistake is made; otherwise we would carry on to immediately > following #define which would cause the compiler to complain that > LIBXL_EXTERNAL_CALLERS_ONLY was redefined. Then the developer > might be tempted to add a #ifndef which would be wrong - it would > leave the affected translation unit unprotected by the new > enforcement regime. So let''s be explicit. > > - Fix the one source of files which violate the above principle, the > output from the idl compiler, by removing the redundant inclusion > of libxl.h from the output. > > In the tree I am using as a base at the time of writing, this new > restriction catches three errors: two in libxl_device_disk_remove and > one in libxl__device_disk_local_detach. To avoid entirely breaking my > build I have also done this: > > - Temporarily change -Werror to -Wno-error in the libxl Makefile. > > This patch should not be applied in this form. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> > Cc: Roger Pau Monne <roger.pau@citrix.com> > Cc: Ian Campbell <ian.campbell@citrix.com> > --- > tools/libxl/Makefile | 2 +- > tools/libxl/gentypes.py | 1 - > tools/libxl/libxl.h | 34 +++++++++++++++++++++++++--------- > tools/libxl/libxl_event.h | 21 ++++++++++++++------- > tools/libxl/libxl_internal.h | 14 ++++++++++---- > 5 files changed, 50 insertions(+), 22 deletions(-) > > diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile > index ddc2624..c238ac0 100644 > --- a/tools/libxl/Makefile > +++ b/tools/libxl/Makefile > @@ -11,7 +11,7 @@ MINOR = 0 > XLUMAJOR = 1.0 > XLUMINOR = 0 > > -CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \ > +CFLAGS += -Wno-error -Wno-format-zero-length -Wmissing-declarations \ > -Wno-declaration-after-statement -Wformat-nonliteral > CFLAGS += -I. -fPIC > > diff --git a/tools/libxl/gentypes.py b/tools/libxl/gentypes.py > index 3c561ba..6e83b21 100644 > --- a/tools/libxl/gentypes.py > +++ b/tools/libxl/gentypes.py > @@ -341,7 +341,6 @@ if __name__ == ''__main__'': > #include <stdlib.h> > #include <string.h> > > -#include "libxl.h" > #include "libxl_internal.h" > > #define LIBXL_DTOR_POISON 0xa5 > diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h > index 10d7115..1a32d9e 100644 > --- a/tools/libxl/libxl.h > +++ b/tools/libxl/libxl.h > @@ -266,6 +266,13 @@ > #endif > #endif > > +/* Functions annotated with LIBXL_EXTERNAL_CALLERS_ONLY may not be > + * called from within libxl itself. Callers outside libxl, who > + * do not #include libxl_internal.h, are fine. */ > +#ifndef LIBXL_EXTERNAL_CALLERS_ONLY > +#define LIBXL_EXTERNAL_CALLERS_ONLY /* disappears for callers outside libxl */ > +#endif > + > typedef uint8_t libxl_mac[6]; > #define LIBXL_MAC_FMT "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx" > #define LIBXL_MAC_FMTLEN ((2*6)+5) /* 6 hex bytes plus 5 colons */ > @@ -495,11 +502,13 @@ int libxl_ctx_free(libxl_ctx *ctx /* 0 is OK */); > int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config, > uint32_t *domid, > const libxl_asyncop_how *ao_how, > - const libxl_asyncprogress_how *aop_console_how); > + const libxl_asyncprogress_how *aop_console_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config, > uint32_t *domid, int restore_fd, > const libxl_asyncop_how *ao_how, > - const libxl_asyncprogress_how *aop_console_how); > + const libxl_asyncprogress_how *aop_console_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > /* A progress report will be made via ao_console_how, of type > * domain_create_console_available, when the domain''s primary > * console is available and can be connected to. > @@ -510,7 +519,8 @@ void libxl_domain_config_dispose(libxl_domain_config *d_config); > > int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, > int flags, /* LIBXL_SUSPEND_* */ > - const libxl_asyncop_how *ao_how); > + const libxl_asyncop_how *ao_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > #define LIBXL_SUSPEND_DEBUG 1 > #define LIBXL_SUSPEND_LIVE 2 > > @@ -522,7 +532,8 @@ int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int suspend_cancel); > > int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info, > uint32_t domid, int send_fd, int recv_fd, > - const libxl_asyncop_how *ao_how); > + const libxl_asyncop_how *ao_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > > int libxl_domain_shutdown(libxl_ctx *ctx, uint32_t domid); > int libxl_domain_reboot(libxl_ctx *ctx, uint32_t domid); > @@ -544,7 +555,8 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid); > > int libxl_domain_core_dump(libxl_ctx *ctx, uint32_t domid, > const char *filename, > - const libxl_asyncop_how *ao_how); > + const libxl_asyncop_how *ao_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > > int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint32_t target_memkb); > int libxl_set_memory_target(libxl_ctx *ctx, uint32_t domid, int32_t target_memkb, int relative, int enforce); > @@ -653,7 +665,8 @@ void libxl_vminfo_list_free(libxl_vminfo *list, int nr); > int libxl_device_disk_add(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk); > int libxl_device_disk_remove(libxl_ctx *ctx, uint32_t domid, > libxl_device_disk *disk, > - const libxl_asyncop_how *ao_how); > + const libxl_asyncop_how *ao_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > int libxl_device_disk_destroy(libxl_ctx *ctx, uint32_t domid, > libxl_device_disk *disk); > > @@ -671,7 +684,8 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk); > int libxl_device_nic_add(libxl_ctx *ctx, uint32_t domid, libxl_device_nic *nic); > int libxl_device_nic_remove(libxl_ctx *ctx, uint32_t domid, > libxl_device_nic *nic, > - const libxl_asyncop_how *ao_how); > + const libxl_asyncop_how *ao_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > int libxl_device_nic_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_nic *nic); > > libxl_device_nic *libxl_device_nic_list(libxl_ctx *ctx, uint32_t domid, int *num); > @@ -682,14 +696,16 @@ int libxl_device_nic_getinfo(libxl_ctx *ctx, uint32_t domid, > int libxl_device_vkb_add(libxl_ctx *ctx, uint32_t domid, libxl_device_vkb *vkb); > int libxl_device_vkb_remove(libxl_ctx *ctx, uint32_t domid, > libxl_device_vkb *vkb, > - const libxl_asyncop_how *ao_how); > + const libxl_asyncop_how *ao_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > int libxl_device_vkb_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_vkb *vkb); > > /* Framebuffer */ > int libxl_device_vfb_add(libxl_ctx *ctx, uint32_t domid, libxl_device_vfb *vfb); > int libxl_device_vfb_remove(libxl_ctx *ctx, uint32_t domid, > libxl_device_vfb *vfb, > - const libxl_asyncop_how *ao_how); > + const libxl_asyncop_how *ao_how) > + LIBXL_EXTERNAL_CALLERS_ONLY; > int libxl_device_vfb_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_vfb *vfb); > > /* PCI Passthrough */ > diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h > index 713d96d..3344bc8 100644 > --- a/tools/libxl/libxl_event.h > +++ b/tools/libxl/libxl_event.h > @@ -37,7 +37,8 @@ typedef int libxl_event_predicate(const libxl_event*, void *user); > > int libxl_event_check(libxl_ctx *ctx, libxl_event **event_r, > uint64_t typemask, > - libxl_event_predicate *predicate, void *predicate_user); > + libxl_event_predicate *predicate, void *predicate_user) > + LIBXL_EXTERNAL_CALLERS_ONLY; > /* Searches for an event, already-happened, which matches typemask > * and predicate. predicate==0 matches any event. > * libxl_event_check returns the event, which must then later be > @@ -48,7 +49,8 @@ int libxl_event_check(libxl_ctx *ctx, libxl_event **event_r, > > int libxl_event_wait(libxl_ctx *ctx, libxl_event **event_r, > uint64_t typemask, > - libxl_event_predicate *predicate, void *predicate_user); > + libxl_event_predicate *predicate, void *predicate_user) > + LIBXL_EXTERNAL_CALLERS_ONLY; > /* Like libxl_event_check but blocks if no suitable events are > * available, until some are. Uses libxl_osevent_beforepoll/ > * _afterpoll so may be inefficient if very many domains are being > @@ -256,7 +258,8 @@ struct pollfd; > */ > int libxl_osevent_beforepoll(libxl_ctx *ctx, int *nfds_io, > struct pollfd *fds, int *timeout_upd, > - struct timeval now); > + struct timeval now) > + LIBXL_EXTERNAL_CALLERS_ONLY; > > /* nfds and fds[0..nfds] must be from the most recent call to > * _beforepoll, as modified by poll. (It is therefore not possible > @@ -271,7 +274,8 @@ int libxl_osevent_beforepoll(libxl_ctx *ctx, int *nfds_io, > * libxl_event_check. > */ > void libxl_osevent_afterpoll(libxl_ctx *ctx, int nfds, const struct pollfd *fds, > - struct timeval now); > + struct timeval now) > + LIBXL_EXTERNAL_CALLERS_ONLY; > > > typedef struct libxl_osevent_hooks { > @@ -357,14 +361,16 @@ void libxl_osevent_register_hooks(libxl_ctx *ctx, > */ > > void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl, > - int fd, short events, short revents); > + int fd, short events, short revents) > + LIBXL_EXTERNAL_CALLERS_ONLY; > > /* Implicitly, on entry to this function the timeout has been > * deregistered. If _occurred_timeout is called, libxl will not > * call timeout_deregister; if it wants to requeue the timeout it > * will call timeout_register again. > */ > -void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl); > +void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl) > + LIBXL_EXTERNAL_CALLERS_ONLY; > > > /*======================================================================*/ > @@ -506,7 +512,8 @@ void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks, > * certainly need to use the self-pipe trick (or a working pselect or > * ppoll) to implement this. > */ > -int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status); > +int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status) > + LIBXL_EXTERNAL_CALLERS_ONLY; > > > /* > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index 36c75ed..6c859bc 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -52,6 +52,12 @@ > > #include <xen/io/xenbus.h> > > +#ifdef LIBXL_H > +# error libxl.h should be included via libxl_internal.h, not separately > +#endif > +#define LIBXL_EXTERNAL_CALLERS_ONLY \ > + __attribute__((warning("may not be called from within libxl"))) > + > #include "libxl.h" > #include "_paths.h" > #include "_libxl_save_msgs_callout.h" > @@ -1538,10 +1544,10 @@ libxl__device_model_version_running(libxl__gc *gc, uint32_t domid); > * > * Functions using LIBXL__INIT_EGC may *not* generally be called from > * within libxl, because libxl__egc_cleanup may call back into the > - * application. This should be documented near the function > - * prototype(s) for callers of LIBXL__INIT_EGC and EGC_INIT. You > - * should in any case not find it necessary to call egc-creators from > - * within libxl. > + * application. This should be enforced by declaring all such > + * functions in libxl.h or libxl_event.h with > + * LIBXL_EXTERNAL_CALLERS_ONLY. You should in any case not find it > + * necessary to call egc-creators from within libxl. > * > * The callbacks must all take place with the ctx unlocked because > * the application is entitled to reenter libxl from them. This > -- > 1.7.2.5 >
Ian Campbell
2012-Jun-22 11:47 UTC
Re: [PATCH 05/21] libxl: domain save: API changes for asynchrony
> @@ -648,32 +648,51 @@ libxl_vminfo * libxl_list_vm(libxl_ctx *ctx, int *nb_vm) > return ptr; > } > > +static void remus_crashed_cb(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc);I think you were going to rename this remus_failover_cb but nevertheless: Acked-by: Ian Campbell <ian.campbell@citrix.com> Were you going to ping Shriram to test remus with this series applied?> + > /* TODO: Explicit Checkpoint acknowledgements via recv_fd. */ > int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info, > - uint32_t domid, int send_fd, int recv_fd) > + uint32_t domid, int send_fd, int recv_fd, > + const libxl_asyncop_how *ao_how) > { > - GC_INIT(ctx); > - libxl_domain_type type = libxl__domain_type(gc, domid); > - int rc = 0; > + AO_CREATE(ctx, domid, ao_how); > + libxl__domain_suspend_state *dss; > + int rc; > > + libxl_domain_type type = libxl__domain_type(gc, domid); > if (type == LIBXL_DOMAIN_TYPE_INVALID) { > rc = ERROR_FAIL; > - goto remus_fail; > + goto out; > } > > - if (info == NULL) { > - LIBXL__LOG(ctx, LIBXL__LOG_ERROR, > - "No remus_info structure supplied for domain %d", domid); > - rc = ERROR_INVAL; > - goto remus_fail; > - } > + GCNEW(dss); > + dss->ao = ao; > + dss->callback = remus_crashed_cb; > + dss->domid = domid; > + dss->fd = send_fd; > + /* TODO do something with recv_fd */ > + dss->type = type; > + dss->live = 1; > + dss->debug = 0; > + dss->remus = info; > + > + assert(info); > > /* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */ > > /* Point of no return */ > - rc = libxl__domain_suspend_common(gc, domid, send_fd, type, /* live */ 1, > - /* debug */ 0, info); > + libxl__domain_suspend(egc, dss); > + return AO_INPROGRESS; > + > + out: > + return AO_ABORT(rc); > +} > > +static void remus_crashed_cb(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc) > +{ > + STATE_AO_GC(dss->ao); > /* > * With Remus, if we reach this point, it means either > * backup died or some network error occurred preventing us > @@ -683,27 +702,46 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info, > /* TBD: Remus cleanup - i.e. detach qdisc, release other > * resources. > */ > - remus_fail: > - GC_FREE; > - return rc; > + libxl__ao_complete(egc, ao, rc); > } > > -int libxl_domain_suspend(libxl_ctx *ctx, libxl_domain_suspend_info *info, > - uint32_t domid, int fd) > +static void domain_suspend_cb(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc) > { > - GC_INIT(ctx); > + STATE_AO_GC(dss->ao); > + libxl__ao_complete(egc,ao,rc); > + > +} > + > +int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags, > + const libxl_asyncop_how *ao_how) > +{ > + AO_CREATE(ctx, domid, ao_how); > + int rc; > + > libxl_domain_type type = libxl__domain_type(gc, domid); > - int live = info != NULL && info->flags & XL_SUSPEND_LIVE; > - int debug = info != NULL && info->flags & XL_SUSPEND_DEBUG; > - int rc = 0; > + if (type == LIBXL_DOMAIN_TYPE_INVALID) { > + rc = ERROR_FAIL; > + goto out_err; > + } > > - rc = libxl__domain_suspend_common(gc, domid, fd, type, live, debug, > - /* No Remus */ NULL); > + libxl__domain_suspend_state *dss; > + GCNEW(dss); > > - if (!rc && type == LIBXL_DOMAIN_TYPE_HVM) > - rc = libxl__domain_save_device_model(gc, domid, fd); > - GC_FREE; > - return rc; > + dss->ao = ao; > + dss->callback = domain_suspend_cb; > + > + dss->domid = domid; > + dss->fd = fd; > + dss->type = type; > + dss->live = flags & LIBXL_SUSPEND_LIVE; > + dss->debug = flags & LIBXL_SUSPEND_DEBUG; > + > + libxl__domain_suspend(egc, dss); > + return AO_INPROGRESS; > + > + out_err: > + return AO_ABORT(rc); > } > > int libxl_domain_pause(libxl_ctx *ctx, uint32_t domid) > diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h > index 05f0e01..10d7115 100644 > --- a/tools/libxl/libxl.h > +++ b/tools/libxl/libxl.h > @@ -347,13 +347,6 @@ typedef struct libxl__ctx libxl_ctx; > > const libxl_version_info* libxl_get_version_info(libxl_ctx *ctx); > > -typedef struct { > -#define XL_SUSPEND_DEBUG 1 > -#define XL_SUSPEND_LIVE 2 > - int flags; > - int (*suspend_callback)(void *, int); > -} libxl_domain_suspend_info; > - > enum { > ERROR_NONSPECIFIC = -1, > ERROR_VERSION = -2, > @@ -514,16 +507,23 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config, > > void libxl_domain_config_init(libxl_domain_config *d_config); > void libxl_domain_config_dispose(libxl_domain_config *d_config); > -int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info, > - uint32_t domid, int send_fd, int recv_fd); > -int libxl_domain_suspend(libxl_ctx *ctx, libxl_domain_suspend_info *info, > - uint32_t domid, int fd); > + > +int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, > + int flags, /* LIBXL_SUSPEND_* */ > + const libxl_asyncop_how *ao_how); > +#define LIBXL_SUSPEND_DEBUG 1 > +#define LIBXL_SUSPEND_LIVE 2 > > /* @param suspend_cancel [from xenctrl.h:xc_domain_resume( @param fast )] > * If this parameter is true, use co-operative resume. The guest > * must support this. > */ > int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int suspend_cancel); > + > +int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info, > + uint32_t domid, int send_fd, int recv_fd, > + const libxl_asyncop_how *ao_how); > + > int libxl_domain_shutdown(libxl_ctx *ctx, uint32_t domid); > int libxl_domain_reboot(libxl_ctx *ctx, uint32_t domid); > int libxl_domain_destroy(libxl_ctx *ctx, uint32_t domid); > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index 1d4e809..b48452c 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -17,13 +17,11 @@ > > #include <glob.h> > > -#include <xenctrl.h> > -#include <xc_dom.h> > +#include "libxl_internal.h" > > +#include <xc_dom.h> > #include <xen/hvm/hvm_info_table.h> > > -#include "libxl_internal.h" > - > libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid) > { > libxl_ctx *ctx = libxl__gc_owner(gc); > @@ -521,11 +519,18 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > return 0; > } > > -static int libxl__domain_suspend_common_switch_qemu_logdirty > +/*==================== Domain suspend (save) ====================*/ > + > +static void domain_suspend_done(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc); > + > +/*----- callbacks, called by xc_domain_save -----*/ > + > +int libxl__domain_suspend_common_switch_qemu_logdirty > (int domid, unsigned int enable, void *data) > { > libxl__domain_suspend_state *dss = data; > - libxl__gc *gc = dss->gc; > + STATE_AO_GC(dss->ao); > char *path; > bool rc; > > @@ -590,10 +595,10 @@ int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid) > return 0; > } > > -static int libxl__domain_suspend_common_callback(void *data) > +int libxl__domain_suspend_common_callback(void *data) > { > libxl__domain_suspend_state *dss = data; > - libxl__gc *gc = dss->gc; > + STATE_AO_GC(dss->ao); > unsigned long hvm_s_state = 0, hvm_pvdrv = 0; > int ret; > char *state = "suspend"; > @@ -714,7 +719,7 @@ static int libxl__domain_suspend_common_callback(void *data) > > guest_suspended: > if (dss->hvm) { > - ret = libxl__domain_suspend_device_model(dss->gc, dss->domid); > + ret = libxl__domain_suspend_device_model(gc, dss->domid); > if (ret) { > LOG(ERROR, "libxl__domain_suspend_device_model failed ret=%d", ret); > return 0; > @@ -731,11 +736,11 @@ static inline char *save_helper(libxl__gc *gc, uint32_t domid, > domid, phys_offset, node); > } > > -static int libxl__toolstack_save(uint32_t domid, uint8_t **buf, > +int libxl__toolstack_save(uint32_t domid, uint8_t **buf, > uint32_t *len, void *data) > { > libxl__domain_suspend_state *dss = data; > - libxl__gc *gc = dss->gc; > + STATE_AO_GC(dss->ao); > int i = 0; > char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL; > unsigned int num = 0; > @@ -806,6 +811,8 @@ static int libxl__toolstack_save(uint32_t domid, uint8_t **buf, > return 0; > } > > +/*----- remus callbacks -----*/ > + > static int libxl__remus_domain_suspend_callback(void *data) > { > /* TODO: Issue disk and network checkpoint reqs. */ > @@ -815,7 +822,7 @@ static int libxl__remus_domain_suspend_callback(void *data) > static int libxl__remus_domain_resume_callback(void *data) > { > libxl__domain_suspend_state *dss = data; > - libxl__gc *gc = dss->gc; > + STATE_AO_GC(dss->ao); > > /* Resumes the domain and the device model */ > if (libxl_domain_resume(CTX, dss->domid, /* Fast Suspend */1)) > @@ -828,10 +835,11 @@ static int libxl__remus_domain_resume_callback(void *data) > static int libxl__remus_domain_checkpoint_callback(void *data) > { > libxl__domain_suspend_state *dss = data; > + STATE_AO_GC(dss->ao); > > /* This would go into tailbuf. */ > if (dss->hvm && > - libxl__domain_save_device_model(dss->gc, dss->domid, dss->save_fd)) > + libxl__domain_save_device_model(gc, dss->domid, dss->fd)) > return 0; > > /* TODO: Wait for disk and memory ack, release network buffer */ > @@ -839,17 +847,23 @@ static int libxl__remus_domain_checkpoint_callback(void *data) > return 1; > } > > -int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > - libxl_domain_type type, > - int live, int debug, > - const libxl_domain_remus_info *r_info) > +/*----- main code for suspending, in order of execution -----*/ > + > +void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss) > { > + STATE_AO_GC(dss->ao); > int port; > - struct save_callbacks callbacks[1]; > - libxl__domain_suspend_state dss[1]; > int rc = ERROR_FAIL; > unsigned long vm_generationid_addr; > > + /* Convenience aliases */ > + const uint32_t domid = dss->domid; > + const libxl_domain_type type = dss->type; > + const int live = dss->live; > + const int debug = dss->debug; > + const libxl_domain_remus_info *const r_info = dss->remus; > + struct save_callbacks *const callbacks = &dss->callbacks; > + > switch (type) { > case LIBXL_DOMAIN_TYPE_HVM: { > char *path; > @@ -868,15 +882,13 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > dss->hvm = 0; > break; > default: > - return ERROR_INVAL; > + abort(); > } > > dss->xcflags = (live) ? XCFLAGS_LIVE : 0 > | (debug) ? XCFLAGS_DEBUG : 0 > | (dss->hvm) ? XCFLAGS_HVM : 0; > > - dss->domid = domid; > - dss->gc = gc; > dss->suspend_eventchn = -1; > dss->guest_responded = 0; > > @@ -884,10 +896,7 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > dss->interval = r_info->interval; > if (r_info->compression) > dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS; > - dss->save_fd = fd; > } > - else > - dss->save_fd = -1; > > dss->xce = xc_evtchn_open(NULL, 0); > if (dss->xce == NULL) > @@ -917,10 +926,28 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > callbacks->toolstack_save = libxl__toolstack_save; > callbacks->data = dss; > > - rc = xc_domain_save(CTX->xch, fd, domid, 0, 0, dss->xcflags, callbacks, > - dss->hvm, vm_generationid_addr); > - if ( rc ) { > - LOGE(ERROR, "saving domain: %s", > + libxl__xc_domain_save(egc, dss, vm_generationid_addr); > + return; > + > + out: > + domain_suspend_done(egc, dss, rc); > +} > + > +void libxl__xc_domain_save_done(libxl__egc *egc, > + libxl__domain_suspend_state *dss, > + int rc, int retval, int errnoval) > +{ > + STATE_AO_GC(dss->ao); > + > + /* Convenience aliases */ > + const libxl_domain_type type = dss->type; > + const uint32_t domid = dss->domid; > + > + if (rc) > + goto out; > + > + if (retval) { > + LOGEV(ERROR, errnoval, "saving domain: %s", > dss->guest_responded ? > "domain responded to suspend request" : > "domain did not respond to suspend request"); > @@ -928,16 +955,21 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > rc = ERROR_GUEST_TIMEDOUT; > else > rc = ERROR_FAIL; > + goto out; > } > > - if (dss->suspend_eventchn > 0) > - xc_suspend_evtchn_release(CTX->xch, dss->xce, domid, > - dss->suspend_eventchn); > - if (dss->xce != NULL) > - xc_evtchn_close(dss->xce); > + if (type == LIBXL_DOMAIN_TYPE_HVM) { > + rc = libxl__domain_suspend_device_model(gc, domid); > + if (rc) goto out; > + > + rc = libxl__domain_save_device_model(gc, domid, dss->fd); > + if (rc) goto out; > + } > + > + rc = 0; > > out: > - return rc; > + domain_suspend_done(egc, dss, rc); > } > > int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) > @@ -992,6 +1024,25 @@ out: > return rc; > } > > +static void domain_suspend_done(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc) > +{ > + STATE_AO_GC(dss->ao); > + > + /* Convenience aliases */ > + const uint32_t domid = dss->domid; > + > + if (dss->suspend_eventchn > 0) > + xc_suspend_evtchn_release(CTX->xch, dss->xce, domid, > + dss->suspend_eventchn); > + if (dss->xce != NULL) > + xc_evtchn_close(dss->xce); > + > + dss->callback(egc, dss, rc); > +} > + > +/*==================== Miscellaneous ====================*/ > + > char *libxl__uuid2string(libxl__gc *gc, const libxl_uuid uuid) > { > char *s = libxl__sprintf(gc, LIBXL_UUID_FMT, LIBXL_UUID_BYTES(uuid)); > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index 28478ea..7cf1b04 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -1777,16 +1777,28 @@ _hidden int libxl__datacopier_start(libxl__datacopier_state *dc); > > typedef struct libxl__domain_suspend_state libxl__domain_suspend_state; > > +typedef void libxl__domain_suspend_cb(libxl__egc*, > + libxl__domain_suspend_state*, int rc); > + > struct libxl__domain_suspend_state { > - libxl__gc *gc; > + /* set by caller of libxl__domain_suspend */ > + libxl__ao *ao; > + libxl__domain_suspend_cb *callback; > + > + uint32_t domid; > + int fd; > + libxl_domain_type type; > + int live; > + int debug; > + const libxl_domain_remus_info *remus; > + /* private */ > xc_evtchn *xce; /* event channel handle */ > int suspend_eventchn; > - int domid; > int hvm; > - unsigned int xcflags; > + int xcflags; > int guest_responded; > - int save_fd; /* Migration stream fd (for Remus) */ > int interval; /* checkpoint interval (for Remus) */ > + struct save_callbacks callbacks; > }; > > > @@ -1903,10 +1915,27 @@ struct libxl__domain_create_state { > > /*----- Domain suspend (save) functions -----*/ > > -_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > - libxl_domain_type type, > - int live, int debug, > - const libxl_domain_remus_info *r_info); > +/* calls dss->callback when done */ > +_hidden void libxl__domain_suspend(libxl__egc *egc, > + libxl__domain_suspend_state *dss); > + > + > +/* calls libxl__xc_domain_suspend_done when done */ > +_hidden void libxl__xc_domain_save(libxl__egc*, libxl__domain_suspend_state*, > + unsigned long vm_generationid_addr); > +/* If rc==0 then retval is the return value from xc_domain_save > + * and errnoval is the errno value it provided. > + * If rc!=0, retval and errnoval are undefined. */ > +_hidden void libxl__xc_domain_save_done(libxl__egc*, > + libxl__domain_suspend_state*, > + int rc, int retval, int errnoval); > + > +_hidden int libxl__domain_suspend_common_callback(void *data); > +_hidden int libxl__domain_suspend_common_switch_qemu_logdirty > + (int domid, unsigned int enable, void *data); > +_hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf, > + uint32_t *len, void *data); > + > > /* calls libxl__xc_domain_restore_done when done */ > _hidden void libxl__xc_domain_restore(libxl__egc *egc, > diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c > index 2f8db9f..1b481ab 100644 > --- a/tools/libxl/libxl_save_callout.c > +++ b/tools/libxl/libxl_save_callout.c > @@ -35,3 +35,14 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs, > &state->vm_generationid_addr, &dcs->callbacks); > libxl__xc_domain_restore_done(egc, dcs, 0, r, errno); > } > + > +void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss, > + unsigned long vm_generationid_addr) > +{ > + STATE_AO_GC(dss->ao); > + int r; > + > + r = xc_domain_save(CTX->xch, dss->fd, dss->domid, 0, 0, dss->xcflags, > + &dss->callbacks, dss->hvm, vm_generationid_addr); > + libxl__xc_domain_save_done(egc, dss, 0, r, errno); > +} > diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c > index 3ea95ef..19daa1c 100644 > --- a/tools/libxl/xl_cmdimpl.c > +++ b/tools/libxl/xl_cmdimpl.c > @@ -2846,7 +2846,7 @@ static int save_domain(const char *p, const char *filename, int checkpoint, > > save_domain_core_writeconfig(fd, filename, config_data, config_len); > > - CHK_ERRNO(libxl_domain_suspend(ctx, NULL, domid, fd)); > + CHK_ERRNO(libxl_domain_suspend(ctx, domid, fd, 0, NULL)); > close(fd); > > if (checkpoint) > @@ -3008,7 +3008,6 @@ static void migrate_domain(const char *domain_spec, const char *rune, > pid_t child = -1; > int rc; > int send_fd = -1, recv_fd = -1; > - libxl_domain_suspend_info suspinfo; > char *away_domname; > char rc_buf; > uint8_t *config_data; > @@ -3030,9 +3029,7 @@ static void migrate_domain(const char *domain_spec, const char *rune, > > xtl_stdiostream_adjust_flags(logger, XTL_STDIOSTREAM_HIDE_PROGRESS, 0); > > - memset(&suspinfo, 0, sizeof(suspinfo)); > - suspinfo.flags |= XL_SUSPEND_LIVE; > - rc = libxl_domain_suspend(ctx, &suspinfo, domid, send_fd); > + rc = libxl_domain_suspend(ctx, domid, send_fd, LIBXL_SUSPEND_LIVE, NULL); > if (rc) { > fprintf(stderr, "migration sender: libxl_domain_suspend failed" > " (rc=%d)\n", rc); > @@ -6604,7 +6601,7 @@ int main_remus(int argc, char **argv) > } > > /* Point of no return */ > - rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd); > + rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd, 0); > > /* If we are here, it means backup has failed/domain suspend failed. > * Try to resume the domain and exit gracefully. > -- > 1.7.2.5 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Ian Campbell
2012-Jun-22 11:59 UTC
Re: [PATCH 06/21] libxl: domain save/restore: run in a separate process
> [...] > +/* stream_fd is as from the caller (eventually, the application). > + * It may be 0, 1 or 2, in which case we need to dup it elsewhere. > + * The actual fd value is not included in the supplied argnums; rather > + * it will be automatically supplied by run_helper as the 2nd argument. > + * > + * preserve_fds are fds that the caller is intending to pass to the > + * helper so which need cloexec clearing. They may not be 0, 1 or 2. > + * An entry may be -1 in which case it will be ignored.[...]> +static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs, > + const char *mode_arg, int stream_fd, > + const int *preserve_fds, int num_preserve_fds, > + const unsigned long *argnums, int num_argnums) > +{ > + STATE_AO_GC(shs->ao); > + const char *args[4 + num_argnums]; > + const char **arg = args; > + int i, rc; > + > + /* Resources we must free */ > + libxl__carefd *childs_pipes[2] = { 0,0 }; > + > + /* Convenience aliases */ > + const uint32_t domid = shs->domid; > + > + shs->rc = 0; > + shs->completed = 0; > + shs->pipes[0] = shs->pipes[1] = 0; > + libxl__ev_fd_init(&shs->readable); > + libxl__ev_child_init(&shs->child); > + > + shs->stdin_what = GCSPRINTF("domain %"PRIu32" save/restore helper" > + " stdin pipe", domid); > + shs->stdout_what = GCSPRINTF("domain %"PRIu32" save/restore helper" > + " stdout pipe", domid); > + > + *arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC "/" "libxl-save-helper"; > + *arg++ = mode_arg; > + const char **stream_fd_arg = arg++; > + for (i=0; i<num_argnums; i++) > + *arg++ = GCSPRINTF("%lu", argnums[i]); > + *arg++ = 0; > + assert(arg == args + ARRAY_SIZE(args)); > + > + libxl__carefd_begin(); > + int childfd; > + for (childfd=0; childfd<2; childfd++) { > + /* Setting up the pipe for the child''s fd childfd */ > + int fds[2]; > + if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; } > + int childs_end = childfd==0 ? 0 /*read*/ : 1 /*write*/; > + int our_end = childfd==0 ? 1 /*write*/ : 0 /*read*/; > + childs_pipes[childfd] = libxl__carefd_record(CTX, fds[childs_end]); > + shs->pipes[childfd] = libxl__carefd_record(CTX, fds[our_end]); > + } > + libxl__carefd_unlock(); > + > + pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited); > + if (!pid) { > + if (stream_fd <= 2) { > + stream_fd = dup(stream_fd); > + if (stream_fd < 0) { > + LOGE(ERROR,"dup migration stream fd");LOGE uses CTX -- is that safe after fork, and will it go anywhere anyway?> + exit(-1); > + } > + } > + libxl_fd_set_cloexec(CTX, stream_fd, 0); > + *stream_fd_arg = GCSPRINTF("%d", stream_fd); > + > + for (i=0; i<num_preserve_fds; i++) > + if (preserve_fds[i] >= 0) > + libxl_fd_set_cloexec(CTX, preserve_fds[i], 0);The doc comment says these cannot be 0,1,2. Possibly not worth enforcing though. Other than those two nits: Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell
2012-Jun-22 12:22 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Fri, 2012-06-15 at 12:53 +0100, Ian Jackson wrote:> This is v4 of my series to asyncify save/restore. All comments have > been addressed.Building with this I get: cc1: warnings being treated as errors libxl.c: In function ‘libxl_domain_destroy’: libxl.c:1224: error: ‘dm_present’ may be used uninitialized in this function I expect because for some reason gcc doesn't realise that the switch covers all possible enum values (since the cases all either initialise or jump to out which does not use dm_present). In order that I could continue to test I did this: diff -r fca1330fa367 tools/libxl/libxl.c --- a/tools/libxl/libxl.c Fri Jun 22 13:19:22 2012 +0100 +++ b/tools/libxl/libxl.c Fri Jun 22 13:22:25 2012 +0100 @@ -1244,6 +1244,7 @@ int libxl_domain_destroy(libxl_ctx *ctx, case LIBXL_DOMAIN_TYPE_INVALID: rc = ERROR_FAIL; goto out; + default: abort(); } dom_path = libxl__xs_get_dompath(gc, domid); but you might have a better preference. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Ian Campbell
2012-Jun-22 13:39 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Fri, 2012-06-15 at 14:40 +0100, Stefano Stabellini wrote:> On Fri, 15 Jun 2012, Ian Jackson wrote: > > This is v4 of my series to asyncify save/restore. All comments have > > been addressed. > > > > I have retested this with HVM (RHEL) and PV guests, for both localhost > > migration and save/restore. RHEL seems to have some unreliability > > which occurs with the baseline too. I was not able to test stub dms > > at all because when trying to boot my RHEL guest with a stub dm the dm > > domain immediately crashed. > > IanC, didn''t you test xen-unstable with stubdoms not too long ago? > > In any case we should probably add "working stubdoms" to the blocker list.FWIW I just netbooted my Debian HVM guest with a stubdom on latest tree and it is indeed knackered. Ian.
Ian Campbell
2012-Jun-22 13:49 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Fri, 2012-06-22 at 14:39 +0100, Ian Campbell wrote:> On Fri, 2012-06-15 at 14:40 +0100, Stefano Stabellini wrote: > > On Fri, 15 Jun 2012, Ian Jackson wrote: > > > This is v4 of my series to asyncify save/restore. All comments have > > > been addressed. > > > > > > I have retested this with HVM (RHEL) and PV guests, for both localhost > > > migration and save/restore. RHEL seems to have some unreliability > > > which occurs with the baseline too. I was not able to test stub dms > > > at all because when trying to boot my RHEL guest with a stub dm the dm > > > domain immediately crashed. > > > > IanC, didn''t you test xen-unstable with stubdoms not too long ago? > > > > In any case we should probably add "working stubdoms" to the blocker list. > > FWIW I just netbooted my Debian HVM guest with a stubdom on latest tree > and it is indeed knackered.BTW before I did that I ran it with modern toolstack but whatever old stubdom I had lying around and it was fine so I guess this must be a stubdom side issue for once! I get these logs: xs_watch(/local/domain/0/backend/console/69, be:0x1470e9:69:0x15d3e0) xs_directory(/local/domain/0/backend/console/69): EACCES xs_watch(/local/domain/0/backend/vkbd/69, be:0x14211d:69:0x15d320) xs_directory(/local/domain/0/backend/vkbd/69): EACCES (XEN) event_channel.c:232:d70 EVTCHNOP failure: domain 69, error -22 ERROR: bind_interdomain failed with rc=-22close(0) GPF rip: 0xff9a4, error_code=0 Thread: main RIP: e030:[<00000000000ff9a4>] RSP: e02b:00000000005cf748 EFLAGS: 00010202 RAX: 2f302f6e69616d6f RBX: 0000002002009e20 RCX: 0000000000000e09 RDX: 0000000000000000 RSI: 00000000deadbeef RDI: 2f302f6e69616d6f RBP: 00000000005cf748 R08: 0000000000000608 R09: 000000000000000a R10: 000000000056f000 R11: fefefefefefefeff R12: 0000000000161980 R13: 0000000000000000 R14: 0000000000162590 R15: 0000000000000000 base is 0x5cf748 caller is 0xe587d base is 0x5cf798 caller is 0xe5a00 base is 0x5cf7a8 caller is 0xe2686 base is 0x5cf7c8 caller is 0x106e5d base is 0x5cf7e8 caller is 0xf7b60 base is 0x5cf818 caller is 0xf9b75 base is 0x5cf868 caller is 0xf7ab4 base is 0x5cf888 caller is 0x2b6ee base is 0x5cf958 caller is 0x2c369 base is 0x5cf978 caller is 0x1da01 base is 0x5cf9e8 caller is 0x8cf9 base is 0x5cfdf8 caller is 0xdabcb base is 0x5cffe8 caller is 0x33da Ian.
Ian Jackson
2012-Jun-26 15:41 UTC
Re: [PATCH 21/21] libxl: DO NOT APPLY enforce prohibition on internal
Ian Campbell writes ("Re: [PATCH 21/21] libxl: DO NOT APPLY enforce
prohibition on internal"):> On Fri, 2012-06-15 at 12:54 +0100, Ian Jackson wrote:
> > DO NOT APPLY THIS PATCH.
> > It contains -Wno-error. Without that it would break the build.
>
> I''m happy with the shape of this patch, and apart from the above
caveat
> I''d be happy to Ack it, assuming it were preceded by fixes to
those
> issues.
>
> I fear slightly that we will be continually forgetting to add the tag
> during review. Is there a way we could augment with run time checking?
> Perhaps LIBXL__INIT_EGC can tell if a CTX is not "outer" somehow?
No, I think it can''t without doing a lot of very tiresome things at
runtime with thread-local variables.
It would be possible via a tangle of #defines to arrange that the
_definition_ of such a function would have to have a weird type:
So you would do:
/* libxl.h */
int libxl_device_disk_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk,
const libxl_asyncop_how *ao_how)
LIBXL_EXTERNAL_CALLERS_ONLY;
/* libxl.c */
int libxl_device_disk_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk,
MAGIC_SPEEDBUMP_MACRO_TYPE *ao_how)
{
This would stop you from accidentally writing an entirely new function
which took an ao_how, and just c&p''ing its declaration into
libxl.h.
But there would still be nothing stopping you simply forgetting to add
the LIBXL_EXTERNAL_CALLERS_ONLY to the declaration. That doesn''t seem
worthwhile.
The only other thing would be to write a small Perl script to check
for obvious abuses.
[a bit later]
How about this:
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index c238ac0..1c8b62b 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -74,7 +74,8 @@ LIBXL_OBJS += _libxl_types.o libxl_flask.o
_libxl_types_internal.o
$(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
$(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include $(XEN_ROOT)/tools/config.h
AUTOINCS= libxlu_cfg_y.h libxlu_cfg_l.h _libxl_list.h _paths.h \
- _libxl_save_msgs_callout.h _libxl_save_msgs_helper.h
+ _libxl_save_msgs_callout.h _libxl_save_msgs_helper.h \
+ libxl.api-ok
AUTOSRCS= libxlu_cfg_y.c libxlu_cfg_l.c
AUTOSRCS += _libxl_save_msgs_callout.c _libxl_save_msgs_helper.c
LIBXLU_OBJS = libxlu_cfg_y.o libxlu_cfg_l.o libxlu_cfg.o \
@@ -113,6 +114,15 @@ $(LIBXL_OBJS) $(LIBXLU_OBJS) $(XL_OBJS): $(AUTOINCS)
genpath-target = $(call buildmakevars2file,_paths.h.tmp)
$(eval $(genpath-target))
+libxl.api-ok: check-libxl-api-rules libxl.api-for-check
+ perl $^
+
+%.api-for-check: %.h
+ $(CC) $(CPPFLAGS) $(CFLAGS) $(CFLAGS_$*.o) -c -E $< $(APPEND_CFLAGS) \
+ -DLIBXL_EXTERNAL_CALLERS_ONLY=LIBXL_EXTERNAL_CALLERS_ONLY \
+ >$@.new
+ $(call move-if-changed,$@.new,$@)
+
_paths.h: genpath
sed -e "s/\([^=]*\)=\(.*\)/#define \1 \2/g" $@.tmp >$@.2.tmp
rm -f $@.tmp
@@ -200,7 +210,7 @@ install: all
.PHONY: clean
clean:
$(RM) -f _*.h *.o *.so* *.a $(CLIENTS) $(DEPS)
- $(RM) -f _*.c *.pyc _paths.*.tmp
+ $(RM) -f _*.c *.pyc _paths.*.tmp *.api-for-check
$(RM) -f testidl.c.new testidl.c
# $(RM) -f $(AUTOSRCS) $(AUTOINCS)
diff --git a/tools/libxl/check-libxl-api-rules
b/tools/libxl/check-libxl-api-rules
new file mode 100755
index 0000000..e056573
--- /dev/null
+++ b/tools/libxl/check-libxl-api-rules
@@ -0,0 +1,15 @@
+#!/usr/bin/perl -w
+use strict;
+our $needed=0;
+while (<>) {
+ if (m/libxl_asyncop_how[^;]/) {
+ $needed=1;
+ }
+ if (m/LIBXL_EXTERNAL_CALLERS_ONLY/) {
+ $needed=0;
+ }
+ next unless $needed;
+ if (m/\;/) {
+ die "$ARGV:$.:missing LIBXL_EXTERNAL_CALLERS_ONLY";
+ }
+}
Ian Jackson
2012-Jun-26 16:48 UTC
Re: [PATCH 05/21] libxl: domain save: API changes for asynchrony
Ian Campbell writes ("Re: [Xen-devel] [PATCH 05/21] libxl: domain save: API
changes for asynchrony"):>
> > @@ -648,32 +648,51 @@ libxl_vminfo * libxl_list_vm(libxl_ctx *ctx, int
*nb_vm)
> > return ptr;
> > }
> >
> > +static void remus_crashed_cb(libxl__egc *egc,
> > + libxl__domain_suspend_state *dss, int
rc);
>
> I think you were going to rename this remus_failover_cb but
> nevertheless:
Yes, that''s done in my tree.
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Thanks.
> Were you going to ping Shriram to test remus with this series applied?
That''s a good idea. I will do that on my next repost.
Ian.
Ian Jackson
2012-Jun-26 16:52 UTC
Re: [PATCH 06/21] libxl: domain save/restore: run in a separate process
Ian Campbell writes ("Re: [Xen-devel] [PATCH 06/21] libxl: domain
save/restore: run in a separate process"):> > [...]
> > + pid_t pid = libxl__ev_child_fork(gc, &shs->child,
helper_exited);
> > + if (!pid) {
> > + if (stream_fd <= 2) {
> > + stream_fd = dup(stream_fd);
> > + if (stream_fd < 0) {
> > + LOGE(ERROR,"dup migration stream fd");
>
> LOGE uses CTX -- is that safe after fork, and will it go anywhere
> anyway?
From the doc comment for libxl__ev_child_fork:
* The child should go on to exec (or exit) soon. The child may not
* make any further calls to libxl infrastructure, except for memory
* allocation and logging. If the child needs to use xenstore it
* must open its own xs handle and use it directly, rather than via
* the libxl event machinery.
So this is fine according to the doc comment. And the doc comment is,
I think, justified. The logging might go to some file where we share
the descriptor with the parent, but logs are opened with O_APPEND, so
that''s fine. Likewise if it''s going to syslog, the socket
will be
fine.
That general-purpose xtl loggers need to work in children isn''t
explicitly documented in xentoollog.h but I think it''s fine to leave
it that way.
> > + exit(-1);
> > + }
> > + }
> > + libxl_fd_set_cloexec(CTX, stream_fd, 0);
> > + *stream_fd_arg = GCSPRINTF("%d", stream_fd);
> > +
> > + for (i=0; i<num_preserve_fds; i++)
> > + if (preserve_fds[i] >= 0)
> > + libxl_fd_set_cloexec(CTX, preserve_fds[i], 0);
>
> The doc comment says these cannot be 0,1,2. Possibly not worth enforcing
> though.
If this restriction is violated then things are going to go very badly
wrong, so an assertion is probably a good idea. (This will kill the
migration child, not the parent.)
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Thanks,
Ian.
Ian Jackson
2012-Jun-26 17:06 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
Ian Campbell writes ("Re: [Xen-devel] [PATCH v3 00/18] libxl: domain
save/restore: run in a separate process"):> On Fri, 2012-06-15 at 12:53 +0100, Ian Jackson wrote:
> > This is v4 of my series to asyncify save/restore. All comments have
> > been addressed.
>
> Building with this I get:
> cc1: warnings being treated as errors
> libxl.c: In function ‘libxl_domain_destroy’:
> libxl.c:1224: error: ‘dm_present’ may be used uninitialized in this
function
>
> I expect because for some reason gcc doesn't realise that the switch
> covers all possible enum values (since the cases all either initialise
> or jump to out which does not use dm_present).
How annoying.
> In order that I could continue to test I did this:
>
> diff -r fca1330fa367 tools/libxl/libxl.c
> --- a/tools/libxl/libxl.c Fri Jun 22 13:19:22 2012 +0100
> +++ b/tools/libxl/libxl.c Fri Jun 22 13:22:25 2012 +0100
> @@ -1244,6 +1244,7 @@ int libxl_domain_destroy(libxl_ctx *ctx,
> case LIBXL_DOMAIN_TYPE_INVALID:
> rc = ERROR_FAIL;
> goto out;
> + default: abort();
> }
>
> dom_path = libxl__xs_get_dompath(gc, domid);
>
> but you might have a better preference.
Thanks, I have done that.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel