Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH v3 00/18] libxl: domain save/restore: run in a separate process
This is v3 of my series to asyncify save/restore, rebased to current tip, retested, and with all comments addressed. In the list below "A" indicates a patch which has been acked sufficiently to go in (assuming its dependencies were to go in too). "*" indicates a new patch in v3. Preparatory work: 01/19 libxc: xc_domain_restore, make toolstack_restore const-correct 02/19 libxl: domain save: rename variables etc. 03/19 libxl: domain restore: reshuffle, preparing for ao 04/19 libxl: domain save: API changes for asynchrony The meat: 05/19 libxl: domain save/restore: run in a separate process Some fixups: A 06/19 libxl: rename libxl_dom:save_helper to physmap_path 07/19 libxl: provide libxl__xs_*_checked and libxl__xs_transaction_* 08/19 libxl: wait for qemu to acknowledge logdirty command Asyncify writing of qemu save file, too: 09/19 libxl: datacopier: provide "prefix data" facility 10/19 libxl: prepare for asynchronous writing of qemu save file 11/19 libxl: Make libxl__domain_save_device_model asynchronous Fix gc_opt handling: * 12/19 libxl: Add a gc to libxl_get_cpu_topology * 13/19 libxl: Do not pass NULL as gc_opt; introduce NOGC * 14/19 libxl: Get compiler to warn about gc_opt==NULL Work on essentially-unrelated bugs: A 15/19 xl: Handle return value from libxl_domain_suspend correctly A 16/19 libxl: do not leak dms->saved_state 17/19 libxl: do not leak spawned middle children A 18/19 libxl: do not leak an event struct on ignored ao progress * 19/19 libxl: DO NOT APPLY enforce prohibition on internal Thanks, Ian.
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 01/19] libxc: xc_domain_restore, make toolstack_restore const-correct
Update the one provider of this callback, in libxl.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v3:
* No longer introduce function pointer typedefs into the libxc API.
---
tools/libxc/xenguest.h | 2 +-
tools/libxl/libxl_dom.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/libxc/xenguest.h b/tools/libxc/xenguest.h
index 91d53f7..707e31c 100644
--- a/tools/libxc/xenguest.h
+++ b/tools/libxc/xenguest.h
@@ -92,7 +92,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
uint32_t max_iter
/* callbacks provided by xc_domain_restore */
struct restore_callbacks {
/* callback to restore toolstack specific data */
- int (*toolstack_restore)(uint32_t domid, uint8_t *buf,
+ int (*toolstack_restore)(uint32_t domid, const uint8_t *buf,
uint32_t size, void* data);
/* to be provided as the last argument to each callback function */
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 10f8c1f..677db1d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -467,13 +467,13 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
domid, phys_offset, node);
}
-static int libxl__toolstack_restore(uint32_t domid, uint8_t *buf,
+static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data)
{
libxl__gc *gc = (libxl__gc *) data;
libxl_ctx *ctx = gc->owner;
int i, ret;
- uint8_t *ptr = buf;
+ const uint8_t *ptr = buf;
uint32_t count = 0, version = 0;
struct libxl__physmap_info* pi;
char *xs_path;
--
1.7.2.5
Preparatory work for making domain suspend asynchronous:
* Rename `struct suspendinfo'' to `libxl__domain_suspend_state''
and move it to libxl_internal.h.
* Rename variables `si'' to `dss''.
* Change the stack-allocated state and callbacks from
struct suspendinfo si;
struct save_callbacks callbacks;
struct restore_callbacks callbacks;
to
libxl__domain_suspend_state dss[1];
struct save_callbacks callbacks[1];
struct restore_callbacks callbacks[1];
so that it may be referred to as a pointer variable everywhere.
* Rename the variable `flags'' (in libxl__domain_suspend_state) to
`xcflags'', to help distinguish it from the other `flags''
which is
passed in from the calling application in libxl_domain_suspend_info.
Abolish the local variable in libxl__domain_suspend_common, as it
can use the one in the dss.
* Move the prototypes of suspend-related functions in libxl_internal.h
to after the definition of the state struct.
* Replace several ctx variables with gc variables and
consequently references to ctx with CTX. Change references
to `dss->gc'' in the functional code to simply `gc''.
* Use LOG* rather than LIBXL__LOG* in a number of places.
* In libxl__domain_save_device_model use `rc'' instead of
`ret''.
* Introduce and use `gc'' and `domid'' in
libxl__domain_suspend_common_callback.
* Wrap some long lines.
* Add an extra pair of parens for clarity in a flag test.
* Remove two pointless casts from void* to a struct*.
No functional change whatsoever.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v3:
* Abolish local variables `xcflags'' and `hvm'' in
libxl__domain_suspend_common; just use dss->xcflags and dss->hvm
instead and hence do not lose some of the changes to xcflags.
Changes in v2:
* Make callbacks into arrays (for pointerisation) too.
* Updated to cope with new remus code.
* Fixed typo in commit message.
---
tools/libxl/libxl_dom.c | 261 ++++++++++++++++++++----------------------
tools/libxl/libxl_internal.h | 30 ++++-
2 files changed, 151 insertions(+), 140 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 677db1d..e90030d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -470,7 +470,7 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data)
{
- libxl__gc *gc = (libxl__gc *) data;
+ libxl__gc *gc = data;
libxl_ctx *ctx = gc->owner;
int i, ret;
const uint8_t *ptr = buf;
@@ -531,7 +531,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
/* read signature */
int rc;
int hvm, pae, superpages;
- struct restore_callbacks callbacks;
+ struct restore_callbacks callbacks[1];
int no_incr_generationid;
switch (info->type) {
case LIBXL_DOMAIN_TYPE_HVM:
@@ -539,8 +539,8 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
superpages = 1;
pae = libxl_defbool_val(info->u.hvm.pae);
no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
- callbacks.toolstack_restore = libxl__toolstack_restore;
- callbacks.data = gc;
+ callbacks->toolstack_restore = libxl__toolstack_restore;
+ callbacks->data = gc;
break;
case LIBXL_DOMAIN_TYPE_PV:
hvm = 0;
@@ -556,7 +556,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
state->store_domid, state->console_port,
&state->console_mfn, state->console_domid,
hvm, pae, superpages, no_incr_generationid,
- &state->vm_generationid_addr,
&callbacks);
+ &state->vm_generationid_addr, callbacks);
if ( rc ) {
LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain");
return ERROR_FAIL;
@@ -564,33 +564,23 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t
domid,
return 0;
}
-struct suspendinfo {
- libxl__gc *gc;
- xc_evtchn *xce; /* event channel handle */
- int suspend_eventchn;
- int domid;
- int hvm;
- unsigned int flags;
- int guest_responded;
- int save_fd; /* Migration stream fd (for Remus) */
- int interval; /* checkpoint interval (for Remus) */
-};
-
-static int libxl__domain_suspend_common_switch_qemu_logdirty(int domid,
unsigned int enable, void *data)
+static int libxl__domain_suspend_common_switch_qemu_logdirty
+ (int domid, unsigned int enable, void *data)
{
- struct suspendinfo *si = data;
- libxl_ctx *ctx = libxl__gc_owner(si->gc);
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
char *path;
bool rc;
- path = libxl__sprintf(si->gc,
"/local/domain/0/device-model/%u/logdirty/cmd", domid);
+ path = libxl__sprintf(gc,
+ "/local/domain/0/device-model/%u/logdirty/cmd",
domid);
if (!path)
return 1;
if (enable)
- rc = xs_write(ctx->xsh, XBT_NULL, path, "enable",
strlen("enable"));
+ rc = xs_write(CTX->xsh, XBT_NULL, path, "enable",
strlen("enable"));
else
- rc = xs_write(ctx->xsh, XBT_NULL, path, "disable",
strlen("disable"));
+ rc = xs_write(CTX->xsh, XBT_NULL, path, "disable",
strlen("disable"));
return rc ? 0 : 1;
}
@@ -645,53 +635,56 @@ int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid)
static int libxl__domain_suspend_common_callback(void *data)
{
- struct suspendinfo *si = data;
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
int ret;
char *state = "suspend";
int watchdog;
- libxl_ctx *ctx = libxl__gc_owner(si->gc);
xs_transaction_t t;
- if (si->hvm) {
- xc_get_hvm_param(ctx->xch, si->domid, HVM_PARAM_CALLBACK_IRQ,
&hvm_pvdrv);
- xc_get_hvm_param(ctx->xch, si->domid, HVM_PARAM_ACPI_S_STATE,
&hvm_s_state);
+ /* Convenience aliases */
+ const uint32_t domid = dss->domid;
+
+ if (dss->hvm) {
+ xc_get_hvm_param(CTX->xch, domid, HVM_PARAM_CALLBACK_IRQ,
&hvm_pvdrv);
+ xc_get_hvm_param(CTX->xch, domid, HVM_PARAM_ACPI_S_STATE,
&hvm_s_state);
}
- if ((hvm_s_state == 0) && (si->suspend_eventchn >= 0)) {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "issuing %s suspend request via
event channel",
- si->hvm ? "PVHVM" : "PV");
- ret = xc_evtchn_notify(si->xce, si->suspend_eventchn);
+ if ((hvm_s_state == 0) && (dss->suspend_eventchn >= 0)) {
+ LOG(DEBUG, "issuing %s suspend request via event channel",
+ dss->hvm ? "PVHVM" : "PV");
+ ret = xc_evtchn_notify(dss->xce, dss->suspend_eventchn);
if (ret < 0) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "xc_evtchn_notify failed
ret=%d", ret);
+ LOG(ERROR, "xc_evtchn_notify failed ret=%d", ret);
return 0;
}
- ret = xc_await_suspend(ctx->xch, si->xce,
si->suspend_eventchn);
+ ret = xc_await_suspend(CTX->xch, dss->xce,
dss->suspend_eventchn);
if (ret < 0) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "xc_await_suspend failed
ret=%d", ret);
+ LOG(ERROR, "xc_await_suspend failed ret=%d", ret);
return 0;
}
- si->guest_responded = 1;
+ dss->guest_responded = 1;
goto guest_suspended;
}
- if (si->hvm && (!hvm_pvdrv || hvm_s_state)) {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Calling xc_domain_shutdown on
HVM domain");
- xc_domain_shutdown(ctx->xch, si->domid, SHUTDOWN_suspend);
+ if (dss->hvm && (!hvm_pvdrv || hvm_s_state)) {
+ LOG(DEBUG, "Calling xc_domain_shutdown on HVM domain");
+ xc_domain_shutdown(CTX->xch, domid, SHUTDOWN_suspend);
/* The guest does not (need to) respond to this sort of request. */
- si->guest_responded = 1;
+ dss->guest_responded = 1;
} else {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "issuing %s suspend request via
XenBus control node",
- si->hvm ? "PVHVM" : "PV");
+ LOG(DEBUG, "issuing %s suspend request via XenBus control
node",
+ dss->hvm ? "PVHVM" : "PV");
- libxl__domain_pvcontrol_write(si->gc, XBT_NULL, si->domid,
"suspend");
+ libxl__domain_pvcontrol_write(gc, XBT_NULL, domid,
"suspend");
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "wait for the guest to
acknowledge suspend request");
+ LOG(DEBUG, "wait for the guest to acknowledge suspend
request");
watchdog = 60;
while (!strcmp(state, "suspend") && watchdog > 0)
{
usleep(100000);
- state = libxl__domain_pvcontrol_read(si->gc, XBT_NULL,
si->domid);
+ state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
if (!state) state = "";
watchdog--;
@@ -707,17 +700,17 @@ static int libxl__domain_suspend_common_callback(void
*data)
* at the last minute.
*/
if (!strcmp(state, "suspend")) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "guest didn''t
acknowledge suspend, cancelling request");
+ LOG(ERROR, "guest didn''t acknowledge suspend,
cancelling request");
retry_transaction:
- t = xs_transaction_start(ctx->xsh);
+ t = xs_transaction_start(CTX->xsh);
- state = libxl__domain_pvcontrol_read(si->gc, t, si->domid);
+ state = libxl__domain_pvcontrol_read(gc, t, domid);
if (!state) state = "";
if (!strcmp(state, "suspend"))
- libxl__domain_pvcontrol_write(si->gc, t, si->domid,
"");
+ libxl__domain_pvcontrol_write(gc, t, domid, "");
- if (!xs_transaction_end(ctx->xsh, t, 0))
+ if (!xs_transaction_end(CTX->xsh, t, 0))
if (errno == EAGAIN)
goto retry_transaction;
@@ -729,27 +722,29 @@ static int libxl__domain_suspend_common_callback(void
*data)
* case we lost the race while cancelling and should continue.
*/
if (!strcmp(state, "suspend")) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "guest didn''t
acknowledge suspend, request cancelled");
+ LOG(ERROR, "guest didn''t acknowledge suspend, request
cancelled");
return 0;
}
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "guest acknowledged suspend
request");
- si->guest_responded = 1;
+ LOG(DEBUG, "guest acknowledged suspend request");
+ dss->guest_responded = 1;
}
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "wait for the guest to
suspend");
+ LOG(DEBUG, "wait for the guest to suspend");
watchdog = 60;
while (watchdog > 0) {
xc_domaininfo_t info;
usleep(100000);
- ret = xc_domain_getinfolist(ctx->xch, si->domid, 1, &info);
- if (ret == 1 && info.domain == si->domid &&
info.flags & XEN_DOMINF_shutdown) {
+ ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);
+ if (ret == 1 && info.domain == domid &&
+ (info.flags & XEN_DOMINF_shutdown)) {
int shutdown_reason;
- shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
& XEN_DOMINF_shutdownmask;
+ shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
+ & XEN_DOMINF_shutdownmask;
if (shutdown_reason == SHUTDOWN_suspend) {
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "guest has
suspended");
+ LOG(DEBUG, "guest has suspended");
goto guest_suspended;
}
}
@@ -757,15 +752,14 @@ static int libxl__domain_suspend_common_callback(void
*data)
watchdog--;
}
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "guest did not suspend");
+ LOG(ERROR, "guest did not suspend");
return 0;
guest_suspended:
- if (si->hvm) {
- ret = libxl__domain_suspend_device_model(si->gc, si->domid);
+ if (dss->hvm) {
+ ret = libxl__domain_suspend_device_model(dss->gc, dss->domid);
if (ret) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
- "libxl__domain_suspend_device_model failed
ret=%d", ret);
+ LOG(ERROR, "libxl__domain_suspend_device_model failed
ret=%d", ret);
return 0;
}
}
@@ -783,9 +777,8 @@ static inline char *save_helper(libxl__gc *gc, uint32_t
domid,
static int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
uint32_t *len, void *data)
{
- struct suspendinfo *si = (struct suspendinfo *) data;
- libxl__gc *gc = (libxl__gc *) si->gc;
- libxl_ctx *ctx = gc->owner;
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
int i = 0;
char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL;
unsigned int num = 0;
@@ -814,21 +807,21 @@ static int libxl__toolstack_save(uint32_t domid, uint8_t
**buf,
char *xs_path;
phys_offset = entries[i];
if (phys_offset == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "phys_offset %d is
NULL", i);
+ LOG(ERROR, "phys_offset %d is NULL", i);
return -1;
}
xs_path = save_helper(gc, domid, phys_offset, "start_addr");
start_addr = libxl__xs_read(gc, 0, xs_path);
if (start_addr == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "%s is NULL", xs_path);
+ LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
xs_path = save_helper(gc, domid, phys_offset, "size");
size = libxl__xs_read(gc, 0, xs_path);
if (size == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "%s is NULL", xs_path);
+ LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
@@ -864,11 +857,11 @@ static int libxl__remus_domain_suspend_callback(void
*data)
static int libxl__remus_domain_resume_callback(void *data)
{
- struct suspendinfo *si = data;
- libxl_ctx *ctx = libxl__gc_owner(si->gc);
+ libxl__domain_suspend_state *dss = data;
+ libxl__gc *gc = dss->gc;
/* Resumes the domain and the device model */
- if (libxl_domain_resume(ctx, si->domid, /* Fast Suspend */1))
+ if (libxl_domain_resume(CTX, dss->domid, /* Fast Suspend */1))
return 0;
/* TODO: Deal with disk. Start a new network output buffer */
@@ -877,15 +870,15 @@ static int libxl__remus_domain_resume_callback(void *data)
static int libxl__remus_domain_checkpoint_callback(void *data)
{
- struct suspendinfo *si = data;
+ libxl__domain_suspend_state *dss = data;
/* This would go into tailbuf. */
- if (si->hvm &&
- libxl__domain_save_device_model(si->gc, si->domid,
si->save_fd))
+ if (dss->hvm &&
+ libxl__domain_save_device_model(dss->gc, dss->domid,
dss->save_fd))
return 0;
/* TODO: Wait for disk and memory ack, release network buffer */
- usleep(si->interval * 1000);
+ usleep(dss->interval * 1000);
return 1;
}
@@ -894,12 +887,10 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
int live, int debug,
const libxl_domain_remus_info *r_info)
{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- int flags;
int port;
- struct save_callbacks callbacks;
- struct suspendinfo si;
- int hvm, rc = ERROR_FAIL;
+ struct save_callbacks callbacks[1];
+ libxl__domain_suspend_state dss[1];
+ int rc = ERROR_FAIL;
unsigned long vm_generationid_addr;
switch (type) {
@@ -912,82 +903,81 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
addr = libxl__xs_read(gc, XBT_NULL, path);
vm_generationid_addr = (addr) ? strtoul(addr, NULL, 0) : 0;
- hvm = 1;
+ dss->hvm = 1;
break;
}
case LIBXL_DOMAIN_TYPE_PV:
vm_generationid_addr = 0;
- hvm = 0;
+ dss->hvm = 0;
break;
default:
return ERROR_INVAL;
}
- memset(&si, 0, sizeof(si));
- flags = (live) ? XCFLAGS_LIVE : 0
+ dss->xcflags = (live) ? XCFLAGS_LIVE : 0
| (debug) ? XCFLAGS_DEBUG : 0
- | (hvm) ? XCFLAGS_HVM : 0;
+ | (dss->hvm) ? XCFLAGS_HVM : 0;
+
+ dss->domid = domid;
+ dss->gc = gc;
+ dss->suspend_eventchn = -1;
+ dss->guest_responded = 0;
if (r_info != NULL) {
- si.interval = r_info->interval;
+ dss->interval = r_info->interval;
if (r_info->compression)
- flags |= XCFLAGS_CHECKPOINT_COMPRESS;
- si.save_fd = fd;
+ dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
+ dss->save_fd = fd;
}
else
- si.save_fd = -1;
+ dss->save_fd = -1;
- si.domid = domid;
- si.flags = flags;
- si.hvm = hvm;
- si.gc = gc;
- si.suspend_eventchn = -1;
- si.guest_responded = 0;
-
- si.xce = xc_evtchn_open(NULL, 0);
- if (si.xce == NULL)
+ dss->xce = xc_evtchn_open(NULL, 0);
+ if (dss->xce == NULL)
goto out;
else
{
- port = xs_suspend_evtchn_port(si.domid);
+ port = xs_suspend_evtchn_port(dss->domid);
if (port >= 0) {
- si.suspend_eventchn = xc_suspend_evtchn_init(ctx->xch, si.xce,
si.domid, port);
+ dss->suspend_eventchn +
xc_suspend_evtchn_init(CTX->xch, dss->xce, dss->domid, port);
- if (si.suspend_eventchn < 0)
- LIBXL__LOG(ctx, LIBXL__LOG_WARNING, "Suspend event channel
initialization failed");
+ if (dss->suspend_eventchn < 0)
+ LOG(WARN, "Suspend event channel initialization
failed");
}
}
- memset(&callbacks, 0, sizeof(callbacks));
+ memset(callbacks, 0, sizeof(*callbacks));
if (r_info != NULL) {
- callbacks.suspend = libxl__remus_domain_suspend_callback;
- callbacks.postcopy = libxl__remus_domain_resume_callback;
- callbacks.checkpoint = libxl__remus_domain_checkpoint_callback;
+ callbacks->suspend = libxl__remus_domain_suspend_callback;
+ callbacks->postcopy = libxl__remus_domain_resume_callback;
+ callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
} else
- callbacks.suspend = libxl__domain_suspend_common_callback;
+ callbacks->suspend = libxl__domain_suspend_common_callback;
- callbacks.switch_qemu_logdirty =
libxl__domain_suspend_common_switch_qemu_logdirty;
- callbacks.toolstack_save = libxl__toolstack_save;
- callbacks.data = &si;
+ callbacks->switch_qemu_logdirty =
libxl__domain_suspend_common_switch_qemu_logdirty;
+ callbacks->toolstack_save = libxl__toolstack_save;
+ callbacks->data = dss;
- rc = xc_domain_save(ctx->xch, fd, domid, 0, 0, flags, &callbacks,
- hvm, vm_generationid_addr);
+ rc = xc_domain_save(CTX->xch, fd, domid, 0, 0, dss->xcflags,
callbacks,
+ dss->hvm, vm_generationid_addr);
if ( rc ) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "saving domain: %s",
- si.guest_responded ?
+ LOGE(ERROR, "saving domain: %s",
+ dss->guest_responded ?
"domain responded to suspend request" :
"domain did not respond to suspend
request");
- if ( !si.guest_responded )
+ if ( !dss->guest_responded )
rc = ERROR_GUEST_TIMEDOUT;
else
rc = ERROR_FAIL;
}
- if (si.suspend_eventchn > 0)
- xc_suspend_evtchn_release(ctx->xch, si.xce, domid,
si.suspend_eventchn);
- if (si.xce != NULL)
- xc_evtchn_close(si.xce);
+ if (dss->suspend_eventchn > 0)
+ xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
+ dss->suspend_eventchn);
+ if (dss->xce != NULL)
+ xc_evtchn_close(dss->xce);
out:
return rc;
@@ -995,8 +985,7 @@ out:
int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd)
{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- int ret, fd2 = -1, c;
+ int rc, fd2 = -1, c;
char buf[1024];
const char *filename = libxl__device_model_savefile(gc, domid);
struct stat st;
@@ -1004,46 +993,46 @@ int libxl__domain_save_device_model(libxl__gc *gc,
uint32_t domid, int fd)
if (stat(filename, &st) < 0)
{
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Unable to stat qemu save
file\n");
- ret = ERROR_FAIL;
+ LOG(ERROR, "Unable to stat qemu save file\n");
+ rc = ERROR_FAIL;
goto out;
}
qemu_state_len = st.st_size;
- LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Qemu state is %d bytes\n",
qemu_state_len);
+ LOG(DEBUG, "Qemu state is %d bytes\n", qemu_state_len);
- ret = libxl_write_exactly(ctx, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE),
+ rc = libxl_write_exactly(CTX, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE),
"saved-state file", "qemu
signature");
- if (ret)
+ if (rc)
goto out;
- ret = libxl_write_exactly(ctx, fd, &qemu_state_len,
sizeof(qemu_state_len),
+ rc = libxl_write_exactly(CTX, fd, &qemu_state_len,
sizeof(qemu_state_len),
"saved-state file", "saved-state
length");
- if (ret)
+ if (rc)
goto out;
fd2 = open(filename, O_RDONLY);
if (fd2 < 0) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Unable to open qemu save
file\n");
+ LOGE(ERROR, "Unable to open qemu save file\n");
goto out;
}
while ((c = read(fd2, buf, sizeof(buf))) != 0) {
if (c < 0) {
if (errno == EINTR)
continue;
- ret = errno;
+ rc = errno;
goto out;
}
- ret = libxl_write_exactly(
- ctx, fd, buf, c, "saved-state file", "qemu
state");
- if (ret)
+ rc = libxl_write_exactly(
+ CTX, fd, buf, c, "saved-state file", "qemu
state");
+ if (rc)
goto out;
}
- ret = 0;
+ rc = 0;
out:
if (fd2 >= 0) close(fd2);
unlink(filename);
- return ret;
+ return rc;
}
char *libxl__uuid2string(libxl__gc *gc, const libxl_uuid uuid)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index fa4c08f..f22bf94 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -786,10 +786,6 @@ _hidden int libxl__domain_restore_common(libxl__gc *gc,
uint32_t domid,
libxl_domain_build_info *info,
libxl__domain_build_state *state,
int fd);
-_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
- libxl_domain_type type,
- int live, int debug,
- const libxl_domain_remus_info
*r_info);
_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
@@ -1778,6 +1774,23 @@ _hidden void
libxl__datacopier_kill(libxl__datacopier_state *dc);
_hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
+/*----- Domain suspend (save) state structure -----*/
+
+typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
+
+struct libxl__domain_suspend_state {
+ libxl__gc *gc;
+ xc_evtchn *xce; /* event channel handle */
+ int suspend_eventchn;
+ int domid;
+ int hvm;
+ unsigned int xcflags;
+ int guest_responded;
+ int save_fd; /* Migration stream fd (for Remus) */
+ int interval; /* checkpoint interval (for Remus) */
+};
+
+
/*----- openpty -----*/
/*
@@ -1888,6 +1901,15 @@ struct libxl__domain_create_state {
* for the non-stubdom device model. */
};
+/*----- Domain suspend (save) functions -----*/
+
+_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
+ libxl_domain_type type,
+ int live, int debug,
+ const libxl_domain_remus_info
*r_info);
+
+
+
/*
* Convenience macros.
*/
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 03/19] libxl: domain restore: reshuffle, preparing for ao
We are going to arrange that libxl, instead of calling
xc_domain_restore, calls a stub function which forks and execs a
helper program, so that restore can be asynchronous rather than
blocking the whole toolstack.
This stub function will be called libxl__xc_domain_restore.
However, its prospective call site is unsuitable for a function which
needs to make a callback, and is buried in two nested single-call-site
functions which are logically part of the domain creation procedure.
So we first abolish those single-call-site functions, integrate their
contents into domain creation in their proper temporal order, and
break out libxl__xc_domain_restore ready for its reimplementation.
No functional change - just the following reorganisation:
* Abolish libxl__domain_restore_common, as it had only one caller.
Move its contents into (what was) domain_restore.
* There is a new stage function domcreate_rebuild_done containing what
used to be the bulk of domcreate_bootloader_done, since
domcreate_bootloader_done now simply starts the restore (or does the
rebuild) and arranges to call the next stage.
* Move the contents of domain_restore into its correct place in the
domain creation sequence. We put it inside
domcreate_bootloader_done, which now either calls
libxl__xc_domain_restore which will call the new function
domcreate_rebuild_done.
* Various general-purpose local variables (`i'' etc.) and convenience
alias variables need to be shuffled about accordingly.
* Consequently libxl__toolstack_restore needs to gain external linkage
as it is now in a different file to its user.
* Move the xc_domain_save callbacks struct from the stack into
libxl__domain_create_state.
In general the moved code remains almost identical. Two returns in
what used to be libxl__domain_restore_common have been changed to set
the return value and "goto out", and the call sites for the abolished
and new functions have been adjusted.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v2:
* Also move the save callbacks
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_create.c | 244 +++++++++++++++++++++++--------------
tools/libxl/libxl_dom.c | 45 +-------
tools/libxl/libxl_internal.h | 19 +++-
tools/libxl/libxl_save_callout.c | 37 ++++++
5 files changed, 206 insertions(+), 140 deletions(-)
create mode 100644 tools/libxl/libxl_save_callout.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index e7d5cc2..1d8b80a 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -67,6 +67,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o
libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o \
+ libxl_save_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 4456ae8..4439367 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -21,7 +21,6 @@
#include "libxl_arch.h"
#include <xc_dom.h>
-#include <xenguest.h>
void libxl_domain_config_init(libxl_domain_config *d_config)
{
@@ -317,89 +316,6 @@ out:
return ret;
}
-static int domain_restore(libxl__gc *gc, libxl_domain_build_info *info,
- uint32_t domid, int fd,
- libxl__domain_build_state *state)
-{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- char **vments = NULL, **localents = NULL;
- struct timeval start_time;
- int i, ret, esave, flags;
-
- ret = libxl__build_pre(gc, domid, info, state);
- if (ret)
- goto out;
-
- ret = libxl__domain_restore_common(gc, domid, info, state, fd);
- if (ret)
- goto out;
-
- gettimeofday(&start_time, NULL);
-
- switch (info->type) {
- case LIBXL_DOMAIN_TYPE_HVM:
- vments = libxl__calloc(gc, 7, sizeof(char *));
- vments[0] = "rtc/timeoffset";
- vments[1] = (info->u.hvm.timeoffset) ? info->u.hvm.timeoffset :
"";
- vments[2] = "image/ostype";
- vments[3] = "hvm";
- vments[4] = "start_time";
- vments[5] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
- break;
- case LIBXL_DOMAIN_TYPE_PV:
- vments = libxl__calloc(gc, 11, sizeof(char *));
- i = 0;
- vments[i++] = "image/ostype";
- vments[i++] = "linux";
- vments[i++] = "image/kernel";
- vments[i++] = (char *) state->pv_kernel.path;
- vments[i++] = "start_time";
- vments[i++] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
- if (state->pv_ramdisk.path) {
- vments[i++] = "image/ramdisk";
- vments[i++] = (char *) state->pv_ramdisk.path;
- }
- if (state->pv_cmdline) {
- vments[i++] = "image/cmdline";
- vments[i++] = (char *) state->pv_cmdline;
- }
- break;
- default:
- ret = ERROR_INVAL;
- goto out;
- }
- ret = libxl__build_post(gc, domid, info, state, vments, localents);
- if (ret)
- goto out;
-
- if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
- ret = asprintf(&state->saved_state,
- XC_DEVICE_MODEL_RESTORE_FILE".%d", domid);
- ret = (ret < 0) ? ERROR_FAIL : 0;
- }
-
-out:
- if (info->type == LIBXL_DOMAIN_TYPE_PV) {
- libxl__file_reference_unmap(&state->pv_kernel);
- libxl__file_reference_unmap(&state->pv_ramdisk);
- }
-
- esave = errno;
-
- flags = fcntl(fd, F_GETFL);
- if (flags == -1) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to get flags on
restore fd");
- } else {
- flags &= ~O_NONBLOCK;
- if (fcntl(fd, F_SETFL, flags) == -1)
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to put restore
fd"
- " back to blocking mode");
- }
-
- errno = esave;
- return ret;
-}
-
int libxl__domain_make(libxl__gc *gc, libxl_domain_create_info *info,
uint32_t *domid)
{
@@ -580,10 +496,13 @@ static void
domcreate_bootloader_console_available(libxl__egc *egc,
static void domcreate_bootloader_done(libxl__egc *egc,
libxl__bootloader_state *bl,
int rc);
-
static void domcreate_console_available(libxl__egc *egc,
libxl__domain_create_state *dcs);
+static void domcreate_rebuild_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int ret);
+
/* Our own function to clean up and call the user''s callback.
* The final call in the sequence. */
static void domcreate_complete(libxl__egc *egc,
@@ -671,20 +590,20 @@ static void domcreate_console_available(libxl__egc *egc,
static void domcreate_bootloader_done(libxl__egc *egc,
libxl__bootloader_state *bl,
- int ret)
+ int rc)
{
libxl__domain_create_state *dcs = CONTAINER_OF(bl, *dcs, bl);
STATE_AO_GC(bl->ao);
- int i;
/* convenience aliases */
const uint32_t domid = dcs->guest_domid;
libxl_domain_config *const d_config = dcs->guest_config;
+ libxl_domain_build_info *const info = &d_config->b_info;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- libxl_ctx *const ctx = CTX;
+ struct restore_callbacks *const callbacks = &dcs->callbacks;
- if (ret) goto error_out;
+ if (rc) domcreate_rebuild_done(egc, dcs, rc);
/* consume bootloader outputs. state->pv_{kernel,ramdisk} have
* been initialised by the bootloader already.
@@ -700,12 +619,153 @@ static void domcreate_bootloader_done(libxl__egc *egc,
dcs->dmss.dm.callback = domcreate_devmodel_started;
dcs->dmss.callback = domcreate_devmodel_started;
- if ( restore_fd >= 0 ) {
- ret = domain_restore(gc, &d_config->b_info, domid, restore_fd,
state);
+ if ( restore_fd < 0 ) {
+ rc = libxl__domain_build(gc, &d_config->b_info, domid, state);
+ domcreate_rebuild_done(egc, dcs, rc);
+ return;
+ }
+
+ /* Restore */
+
+ rc = libxl__build_pre(gc, domid, info, state);
+ if (rc)
+ goto out;
+
+ /* read signature */
+ int hvm, pae, superpages;
+ int no_incr_generationid;
+ switch (info->type) {
+ case LIBXL_DOMAIN_TYPE_HVM:
+ hvm = 1;
+ superpages = 1;
+ pae = libxl_defbool_val(info->u.hvm.pae);
+ no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
+ callbacks->toolstack_restore = libxl__toolstack_restore;
+ callbacks->data = gc;
+ break;
+ case LIBXL_DOMAIN_TYPE_PV:
+ hvm = 0;
+ superpages = 0;
+ pae = 1;
+ no_incr_generationid = 0;
+ break;
+ default:
+ rc = ERROR_INVAL;
+ goto out;
+ }
+ libxl__xc_domain_restore(egc, dcs,
+ hvm, pae, superpages, no_incr_generationid);
+ return;
+
+ out:
+ libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0);
+}
+
+void libxl__xc_domain_restore_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int ret, int retval, int errnoval)
+{
+ STATE_AO_GC(dcs->ao);
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ char **vments = NULL, **localents = NULL;
+ struct timeval start_time;
+ int i, esave, flags;
+
+ /* convenience aliases */
+ const uint32_t domid = dcs->guest_domid;
+ libxl_domain_config *const d_config = dcs->guest_config;
+ libxl_domain_build_info *const info = &d_config->b_info;
+ libxl__domain_build_state *const state = &dcs->build_state;
+ const int fd = dcs->restore_fd;
+
+ if (ret)
+ goto out;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "restoring domain");
+ ret = ERROR_FAIL;
+ goto out;
+ }
+
+ gettimeofday(&start_time, NULL);
+
+ switch (info->type) {
+ case LIBXL_DOMAIN_TYPE_HVM:
+ vments = libxl__calloc(gc, 7, sizeof(char *));
+ vments[0] = "rtc/timeoffset";
+ vments[1] = (info->u.hvm.timeoffset) ? info->u.hvm.timeoffset :
"";
+ vments[2] = "image/ostype";
+ vments[3] = "hvm";
+ vments[4] = "start_time";
+ vments[5] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
+ break;
+ case LIBXL_DOMAIN_TYPE_PV:
+ vments = libxl__calloc(gc, 11, sizeof(char *));
+ i = 0;
+ vments[i++] = "image/ostype";
+ vments[i++] = "linux";
+ vments[i++] = "image/kernel";
+ vments[i++] = (char *) state->pv_kernel.path;
+ vments[i++] = "start_time";
+ vments[i++] = libxl__sprintf(gc, "%lu.%02d",
start_time.tv_sec,(int)start_time.tv_usec/10000);
+ if (state->pv_ramdisk.path) {
+ vments[i++] = "image/ramdisk";
+ vments[i++] = (char *) state->pv_ramdisk.path;
+ }
+ if (state->pv_cmdline) {
+ vments[i++] = "image/cmdline";
+ vments[i++] = (char *) state->pv_cmdline;
+ }
+ break;
+ default:
+ ret = ERROR_INVAL;
+ goto out;
+ }
+ ret = libxl__build_post(gc, domid, info, state, vments, localents);
+ if (ret)
+ goto out;
+
+ if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
+ ret = asprintf(&state->saved_state,
+ XC_DEVICE_MODEL_RESTORE_FILE".%d", domid);
+ ret = (ret < 0) ? ERROR_FAIL : 0;
+ }
+
+out:
+ if (info->type == LIBXL_DOMAIN_TYPE_PV) {
+ libxl__file_reference_unmap(&state->pv_kernel);
+ libxl__file_reference_unmap(&state->pv_ramdisk);
+ }
+
+ esave = errno;
+
+ flags = fcntl(fd, F_GETFL);
+ if (flags == -1) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to get flags on
restore fd");
} else {
- ret = libxl__domain_build(gc, &d_config->b_info, domid, state);
+ flags &= ~O_NONBLOCK;
+ if (fcntl(fd, F_SETFL, flags) == -1)
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to put restore
fd"
+ " back to blocking mode");
}
+ errno = esave;
+ domcreate_rebuild_done(egc, dcs, ret);
+}
+
+static void domcreate_rebuild_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int ret)
+{
+ STATE_AO_GC(dcs->ao);
+ int i;
+
+ /* convenience aliases */
+ const uint32_t domid = dcs->guest_domid;
+ libxl_domain_config *const d_config = dcs->guest_config;
+ libxl__domain_build_state *const state = &dcs->build_state;
+ libxl_ctx *const ctx = CTX;
+
if (ret) {
LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "cannot (re-)build domain:
%d", ret);
ret = ERROR_FAIL;
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index e90030d..1d4e809 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -19,7 +19,6 @@
#include <xenctrl.h>
#include <xc_dom.h>
-#include <xenguest.h>
#include <xen/hvm/hvm_info_table.h>
@@ -467,7 +466,7 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
domid, phys_offset, node);
}
-static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
+int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data)
{
libxl__gc *gc = data;
@@ -522,48 +521,6 @@ static int libxl__toolstack_restore(uint32_t domid, const
uint8_t *buf,
return 0;
}
-int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid,
- libxl_domain_build_info *info,
- libxl__domain_build_state *state,
- int fd)
-{
- libxl_ctx *ctx = libxl__gc_owner(gc);
- /* read signature */
- int rc;
- int hvm, pae, superpages;
- struct restore_callbacks callbacks[1];
- int no_incr_generationid;
- switch (info->type) {
- case LIBXL_DOMAIN_TYPE_HVM:
- hvm = 1;
- superpages = 1;
- pae = libxl_defbool_val(info->u.hvm.pae);
- no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
- callbacks->toolstack_restore = libxl__toolstack_restore;
- callbacks->data = gc;
- break;
- case LIBXL_DOMAIN_TYPE_PV:
- hvm = 0;
- superpages = 0;
- pae = 1;
- no_incr_generationid = 0;
- break;
- default:
- return ERROR_INVAL;
- }
- rc = xc_domain_restore(ctx->xch, fd, domid,
- state->store_port, &state->store_mfn,
- state->store_domid, state->console_port,
- &state->console_mfn, state->console_domid,
- hvm, pae, superpages, no_incr_generationid,
- &state->vm_generationid_addr, callbacks);
- if ( rc ) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain");
- return ERROR_FAIL;
- }
- return 0;
-}
-
static int libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data)
{
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index f22bf94..28478ea 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -46,6 +46,7 @@
#include <xenstore.h>
#include <xenctrl.h>
+#include <xenguest.h>
#include "xentoollog.h"
@@ -782,10 +783,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t
domid,
const char *old_name, const char *new_name,
xs_transaction_t trans);
-_hidden int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid,
- libxl_domain_build_info *info,
- libxl__domain_build_state *state,
- int fd);
+_hidden int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
+ uint32_t size, void *data);
_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
@@ -1899,6 +1898,7 @@ struct libxl__domain_create_state {
libxl__stub_dm_spawn_state dmss;
/* If we''re not doing stubdom, we use only dmss.dm,
* for the non-stubdom device model. */
+ struct restore_callbacks callbacks;
};
/*----- Domain suspend (save) functions -----*/
@@ -1908,6 +1908,17 @@ _hidden int libxl__domain_suspend_common(libxl__gc *gc,
uint32_t domid, int fd,
int live, int debug,
const libxl_domain_remus_info
*r_info);
+/* calls libxl__xc_domain_restore_done when done */
+_hidden void libxl__xc_domain_restore(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int hvm, int pae, int superpages,
+ int no_incr_generationid);
+/* If rc==0 then retval is the return value from xc_domain_save
+ * and errnoval is the errno value it provided.
+ * If rc!=0, retval and errnoval are undefined. */
+_hidden void libxl__xc_domain_restore_done(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int rc, int retval, int errnoval);
/*
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
new file mode 100644
index 0000000..2f8db9f
--- /dev/null
+++ b/tools/libxl/libxl_save_callout.c
@@ -0,0 +1,37 @@
+/*
+ * Copyright (C) 2012 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h"
+
+#include "libxl_internal.h"
+
+void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
+ int hvm, int pae, int superpages,
+ int no_incr_generationid)
+{
+ STATE_AO_GC(dcs->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = dcs->guest_domid;
+ const int restore_fd = dcs->restore_fd;
+ libxl__domain_build_state *const state = &dcs->build_state;
+
+ int r = xc_domain_restore(CTX->xch, restore_fd, domid,
+ state->store_port, &state->store_mfn,
+ state->store_domid, state->console_port,
+ &state->console_mfn,
state->console_domid,
+ hvm, pae, superpages, no_incr_generationid,
+ &state->vm_generationid_addr,
&dcs->callbacks);
+ libxl__xc_domain_restore_done(egc, dcs, 0, r, errno);
+}
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 04/19] libxl: domain save: API changes for asynchrony
Change the internal and external APIs for domain save (suspend) to be
capable of asynchronous operation. The implementation remains
synchronous. The interfaces surrounding device model saving are still
synchronous.
Public API changes:
* libxl_domain_save takes an ao_how.
* libxl_domain_remus_start takes an ao_how. If the
libxl_domain_remus_info is NULL, we abort rather than returning an
error.
* The `suspend_callback'' function passed to libxl_domain_save is
never called by the existing implementation in libxl. Abolish it.
* libxl_domain_save takes its flags parameter as an argument.
Thus libxl_domain_suspend_info is abolished.
* XL_SUSPEND_* flags renamed to LIBXL_SAVE_*.
* Callers in xl updated.
Internal code restructuring:
* libxl__domain_suspend_state member types and names rationalised.
* libxl__domain_suspend renamed from libxl__domain_suspend_common.
(_common here actually meant "internal function").
* libxl__domain_suspend takes a libxl__domain_suspend_state, which
where the parameters to the operation are filled in by the caller.
* xc_domain_save is now called via libxl__xc_domain_save which can
itself become asynchronous.
* Consequently, libxl__domain_suspend is split into two functions at
the callback boundary; the second half is
libxl__xc_domain_save_done.
* libxl__domain_save_device_model is now called by the actual
implementation rather than by the public wrapper. It is already in
its proper place in the domain save execution sequence. So
officially make it part of that execution sequence, renaming it to
domain_save_device_model.
* Effectively, rewrite the public wrapper functions
libxl_domain_suspend and libxl_domain_remus_start.
* Remove a needless #include <xenctrl.h>
* libxl__domain_suspend aborts on unexpected domain types rather
than mysteriously returning EINVAL.
* struct save_callbacks moved from the stack to the dss.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v3:
* Remove `hvm'' and `xcflags'' args to libxl__xc_domain_save.
Instead,
just use the values from the dss.
Changes in v2:
* Move save_callbacks too.
* Merge with Remus changes.
* Improvements to commit message.
* Do not rename libxl_domain_suspend any more.
---
tools/libxl/libxl.c | 95 +++++++++++++++++++++---------
tools/libxl/libxl.h | 22 ++++----
tools/libxl/libxl_dom.c | 121 +++++++++++++++++++++++++++-----------
tools/libxl/libxl_internal.h | 45 ++++++++++++---
tools/libxl/libxl_save_callout.c | 11 ++++
tools/libxl/xl_cmdimpl.c | 9 +--
6 files changed, 215 insertions(+), 88 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 6215923..7de026b 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -648,32 +648,51 @@ libxl_vminfo * libxl_list_vm(libxl_ctx *ctx, int *nb_vm)
return ptr;
}
+static void remus_crashed_cb(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc);
+
/* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
- uint32_t domid, int send_fd, int recv_fd)
+ uint32_t domid, int send_fd, int recv_fd,
+ const libxl_asyncop_how *ao_how)
{
- GC_INIT(ctx);
- libxl_domain_type type = libxl__domain_type(gc, domid);
- int rc = 0;
+ AO_CREATE(ctx, domid, ao_how);
+ libxl__domain_suspend_state *dss;
+ int rc;
+ libxl_domain_type type = libxl__domain_type(gc, domid);
if (type == LIBXL_DOMAIN_TYPE_INVALID) {
rc = ERROR_FAIL;
- goto remus_fail;
+ goto out;
}
- if (info == NULL) {
- LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
- "No remus_info structure supplied for domain %d",
domid);
- rc = ERROR_INVAL;
- goto remus_fail;
- }
+ GCNEW(dss);
+ dss->ao = ao;
+ dss->callback = remus_crashed_cb;
+ dss->domid = domid;
+ dss->fd = send_fd;
+ /* TODO do something with recv_fd */
+ dss->type = type;
+ dss->live = 1;
+ dss->debug = 0;
+ dss->remus = info;
+
+ assert(info);
/* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */
/* Point of no return */
- rc = libxl__domain_suspend_common(gc, domid, send_fd, type, /* live */ 1,
- /* debug */ 0, info);
+ libxl__domain_suspend(egc, dss);
+ return AO_INPROGRESS;
+
+ out:
+ return AO_ABORT(rc);
+}
+static void remus_crashed_cb(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
+{
+ STATE_AO_GC(dss->ao);
/*
* With Remus, if we reach this point, it means either
* backup died or some network error occurred preventing us
@@ -683,27 +702,47 @@ int libxl_domain_remus_start(libxl_ctx *ctx,
libxl_domain_remus_info *info,
/* TBD: Remus cleanup - i.e. detach qdisc, release other
* resources.
*/
- remus_fail:
- GC_FREE;
- return rc;
+ libxl__ao_complete(egc, ao, rc);
}
-int libxl_domain_suspend(libxl_ctx *ctx, libxl_domain_suspend_info *info,
- uint32_t domid, int fd)
+static void domain_suspend_cb(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
{
- GC_INIT(ctx);
+ STATE_AO_GC(dss->ao);
+ libxl__ao_complete(egc,ao,rc);
+
+}
+
+int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags,
+ const libxl_asyncop_how *ao_how)
+{
+ AO_CREATE(ctx, domid, ao_how);
+ int rc;
+
libxl_domain_type type = libxl__domain_type(gc, domid);
- int live = info != NULL && info->flags & XL_SUSPEND_LIVE;
- int debug = info != NULL && info->flags & XL_SUSPEND_DEBUG;
- int rc = 0;
+ if (type < 0) {
+ LOG(ERROR,"domain %"PRIu32": unable to determine domain
type", domid);
+ rc = ERROR_FAIL;
+ goto out_err;
+ }
- rc = libxl__domain_suspend_common(gc, domid, fd, type, live, debug,
- /* No Remus */ NULL);
+ libxl__domain_suspend_state *dss;
+ GCNEW(dss);
- if (!rc && type == LIBXL_DOMAIN_TYPE_HVM)
- rc = libxl__domain_save_device_model(gc, domid, fd);
- GC_FREE;
- return rc;
+ dss->ao = ao;
+ dss->callback = domain_suspend_cb;
+
+ dss->domid = domid;
+ dss->fd = fd;
+ dss->type = type;
+ dss->live = flags & LIBXL_SUSPEND_LIVE;
+ dss->debug = flags & LIBXL_SUSPEND_DEBUG;
+
+ libxl__domain_suspend(egc, dss);
+ return AO_INPROGRESS;
+
+ out_err:
+ return AO_ABORT(rc);
}
int libxl_domain_pause(libxl_ctx *ctx, uint32_t domid)
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 05f0e01..10d7115 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -347,13 +347,6 @@ typedef struct libxl__ctx libxl_ctx;
const libxl_version_info* libxl_get_version_info(libxl_ctx *ctx);
-typedef struct {
-#define XL_SUSPEND_DEBUG 1
-#define XL_SUSPEND_LIVE 2
- int flags;
- int (*suspend_callback)(void *, int);
-} libxl_domain_suspend_info;
-
enum {
ERROR_NONSPECIFIC = -1,
ERROR_VERSION = -2,
@@ -514,16 +507,23 @@ int libxl_domain_create_restore(libxl_ctx *ctx,
libxl_domain_config *d_config,
void libxl_domain_config_init(libxl_domain_config *d_config);
void libxl_domain_config_dispose(libxl_domain_config *d_config);
-int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
- uint32_t domid, int send_fd, int recv_fd);
-int libxl_domain_suspend(libxl_ctx *ctx, libxl_domain_suspend_info *info,
- uint32_t domid, int fd);
+
+int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd,
+ int flags, /* LIBXL_SUSPEND_* */
+ const libxl_asyncop_how *ao_how);
+#define LIBXL_SUSPEND_DEBUG 1
+#define LIBXL_SUSPEND_LIVE 2
/* @param suspend_cancel [from xenctrl.h:xc_domain_resume( @param fast )]
* If this parameter is true, use co-operative resume. The guest
* must support this.
*/
int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int suspend_cancel);
+
+int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
+ uint32_t domid, int send_fd, int recv_fd,
+ const libxl_asyncop_how *ao_how);
+
int libxl_domain_shutdown(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_reboot(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_destroy(libxl_ctx *ctx, uint32_t domid);
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 1d4e809..b48452c 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -17,13 +17,11 @@
#include <glob.h>
-#include <xenctrl.h>
-#include <xc_dom.h>
+#include "libxl_internal.h"
+#include <xc_dom.h>
#include <xen/hvm/hvm_info_table.h>
-#include "libxl_internal.h"
-
libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid)
{
libxl_ctx *ctx = libxl__gc_owner(gc);
@@ -521,11 +519,18 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t
*buf,
return 0;
}
-static int libxl__domain_suspend_common_switch_qemu_logdirty
+/*==================== Domain suspend (save) ====================*/
+
+static void domain_suspend_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc);
+
+/*----- callbacks, called by xc_domain_save -----*/
+
+int libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
char *path;
bool rc;
@@ -590,10 +595,10 @@ int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid)
return 0;
}
-static int libxl__domain_suspend_common_callback(void *data)
+int libxl__domain_suspend_common_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
int ret;
char *state = "suspend";
@@ -714,7 +719,7 @@ static int libxl__domain_suspend_common_callback(void *data)
guest_suspended:
if (dss->hvm) {
- ret = libxl__domain_suspend_device_model(dss->gc, dss->domid);
+ ret = libxl__domain_suspend_device_model(gc, dss->domid);
if (ret) {
LOG(ERROR, "libxl__domain_suspend_device_model failed
ret=%d", ret);
return 0;
@@ -731,11 +736,11 @@ static inline char *save_helper(libxl__gc *gc, uint32_t
domid,
domid, phys_offset, node);
}
-static int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
+int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
uint32_t *len, void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
int i = 0;
char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL;
unsigned int num = 0;
@@ -806,6 +811,8 @@ static int libxl__toolstack_save(uint32_t domid, uint8_t
**buf,
return 0;
}
+/*----- remus callbacks -----*/
+
static int libxl__remus_domain_suspend_callback(void *data)
{
/* TODO: Issue disk and network checkpoint reqs. */
@@ -815,7 +822,7 @@ static int libxl__remus_domain_suspend_callback(void *data)
static int libxl__remus_domain_resume_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
- libxl__gc *gc = dss->gc;
+ STATE_AO_GC(dss->ao);
/* Resumes the domain and the device model */
if (libxl_domain_resume(CTX, dss->domid, /* Fast Suspend */1))
@@ -828,10 +835,11 @@ static int libxl__remus_domain_resume_callback(void *data)
static int libxl__remus_domain_checkpoint_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
+ STATE_AO_GC(dss->ao);
/* This would go into tailbuf. */
if (dss->hvm &&
- libxl__domain_save_device_model(dss->gc, dss->domid,
dss->save_fd))
+ libxl__domain_save_device_model(gc, dss->domid, dss->fd))
return 0;
/* TODO: Wait for disk and memory ack, release network buffer */
@@ -839,17 +847,23 @@ static int libxl__remus_domain_checkpoint_callback(void
*data)
return 1;
}
-int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
- libxl_domain_type type,
- int live, int debug,
- const libxl_domain_remus_info *r_info)
+/*----- main code for suspending, in order of execution -----*/
+
+void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
{
+ STATE_AO_GC(dss->ao);
int port;
- struct save_callbacks callbacks[1];
- libxl__domain_suspend_state dss[1];
int rc = ERROR_FAIL;
unsigned long vm_generationid_addr;
+ /* Convenience aliases */
+ const uint32_t domid = dss->domid;
+ const libxl_domain_type type = dss->type;
+ const int live = dss->live;
+ const int debug = dss->debug;
+ const libxl_domain_remus_info *const r_info = dss->remus;
+ struct save_callbacks *const callbacks = &dss->callbacks;
+
switch (type) {
case LIBXL_DOMAIN_TYPE_HVM: {
char *path;
@@ -868,15 +882,13 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
dss->hvm = 0;
break;
default:
- return ERROR_INVAL;
+ abort();
}
dss->xcflags = (live) ? XCFLAGS_LIVE : 0
| (debug) ? XCFLAGS_DEBUG : 0
| (dss->hvm) ? XCFLAGS_HVM : 0;
- dss->domid = domid;
- dss->gc = gc;
dss->suspend_eventchn = -1;
dss->guest_responded = 0;
@@ -884,10 +896,7 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
dss->interval = r_info->interval;
if (r_info->compression)
dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
- dss->save_fd = fd;
}
- else
- dss->save_fd = -1;
dss->xce = xc_evtchn_open(NULL, 0);
if (dss->xce == NULL)
@@ -917,10 +926,28 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
callbacks->toolstack_save = libxl__toolstack_save;
callbacks->data = dss;
- rc = xc_domain_save(CTX->xch, fd, domid, 0, 0, dss->xcflags,
callbacks,
- dss->hvm, vm_generationid_addr);
- if ( rc ) {
- LOGE(ERROR, "saving domain: %s",
+ libxl__xc_domain_save(egc, dss, vm_generationid_addr);
+ return;
+
+ out:
+ domain_suspend_done(egc, dss, rc);
+}
+
+void libxl__xc_domain_save_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ int rc, int retval, int errnoval)
+{
+ STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases */
+ const libxl_domain_type type = dss->type;
+ const uint32_t domid = dss->domid;
+
+ if (rc)
+ goto out;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "saving domain: %s",
dss->guest_responded ?
"domain responded to suspend request" :
"domain did not respond to suspend
request");
@@ -928,16 +955,21 @@ int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
rc = ERROR_GUEST_TIMEDOUT;
else
rc = ERROR_FAIL;
+ goto out;
}
- if (dss->suspend_eventchn > 0)
- xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
- dss->suspend_eventchn);
- if (dss->xce != NULL)
- xc_evtchn_close(dss->xce);
+ if (type == LIBXL_DOMAIN_TYPE_HVM) {
+ rc = libxl__domain_suspend_device_model(gc, domid);
+ if (rc) goto out;
+
+ rc = libxl__domain_save_device_model(gc, domid, dss->fd);
+ if (rc) goto out;
+ }
+
+ rc = 0;
out:
- return rc;
+ domain_suspend_done(egc, dss, rc);
}
int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd)
@@ -992,6 +1024,25 @@ out:
return rc;
}
+static void domain_suspend_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
+{
+ STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = dss->domid;
+
+ if (dss->suspend_eventchn > 0)
+ xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
+ dss->suspend_eventchn);
+ if (dss->xce != NULL)
+ xc_evtchn_close(dss->xce);
+
+ dss->callback(egc, dss, rc);
+}
+
+/*==================== Miscellaneous ====================*/
+
char *libxl__uuid2string(libxl__gc *gc, const libxl_uuid uuid)
{
char *s = libxl__sprintf(gc, LIBXL_UUID_FMT, LIBXL_UUID_BYTES(uuid));
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 28478ea..e72b8d2 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1777,16 +1777,28 @@ _hidden int
libxl__datacopier_start(libxl__datacopier_state *dc);
typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
+typedef void libxl__domain_suspend_cb(libxl__egc*,
+ libxl__domain_suspend_state*, int rc);
+
struct libxl__domain_suspend_state {
- libxl__gc *gc;
+ /* set by caller of libxl__domain_suspend */
+ libxl__ao *ao;
+ libxl__domain_suspend_cb *callback;
+
+ uint32_t domid;
+ int fd;
+ libxl_domain_type type;
+ int live;
+ int debug;
+ const libxl_domain_remus_info *remus;
+ /* private */
xc_evtchn *xce; /* event channel handle */
int suspend_eventchn;
- int domid;
int hvm;
- unsigned int xcflags;
+ int xcflags;
int guest_responded;
- int save_fd; /* Migration stream fd (for Remus) */
int interval; /* checkpoint interval (for Remus) */
+ struct save_callbacks callbacks;
};
@@ -1903,10 +1915,27 @@ struct libxl__domain_create_state {
/*----- Domain suspend (save) functions -----*/
-_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd,
- libxl_domain_type type,
- int live, int debug,
- const libxl_domain_remus_info
*r_info);
+/* calls callback when done */
+_hidden void libxl__domain_suspend(libxl__egc *egc,
+ libxl__domain_suspend_state *dss);
+
+
+/* calls libxl__xc_domain_suspend_done when done */
+_hidden void libxl__xc_domain_save(libxl__egc*, libxl__domain_suspend_state*,
+ unsigned long vm_generationid_addr);
+/* If rc==0 then retval is the return value from xc_domain_save
+ * and errnoval is the errno value it provided.
+ * If rc!=0, retval and errnoval are undefined. */
+_hidden void libxl__xc_domain_save_done(libxl__egc*,
+ libxl__domain_suspend_state*,
+ int rc, int retval, int errnoval);
+
+_hidden int libxl__domain_suspend_common_callback(void *data);
+_hidden int libxl__domain_suspend_common_switch_qemu_logdirty
+ (int domid, unsigned int enable, void *data);
+_hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
+ uint32_t *len, void *data);
+
/* calls libxl__xc_domain_restore_done when done */
_hidden void libxl__xc_domain_restore(libxl__egc *egc,
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 2f8db9f..1b481ab 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -35,3 +35,14 @@ void libxl__xc_domain_restore(libxl__egc *egc,
libxl__domain_create_state *dcs,
&state->vm_generationid_addr,
&dcs->callbacks);
libxl__xc_domain_restore_done(egc, dcs, 0, r, errno);
}
+
+void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
+ unsigned long vm_generationid_addr)
+{
+ STATE_AO_GC(dss->ao);
+ int r;
+
+ r = xc_domain_save(CTX->xch, dss->fd, dss->domid, 0, 0,
dss->xcflags,
+ &dss->callbacks, dss->hvm,
vm_generationid_addr);
+ libxl__xc_domain_save_done(egc, dss, 0, r, errno);
+}
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 3ea95ef..19daa1c 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -2846,7 +2846,7 @@ static int save_domain(const char *p, const char
*filename, int checkpoint,
save_domain_core_writeconfig(fd, filename, config_data, config_len);
- CHK_ERRNO(libxl_domain_suspend(ctx, NULL, domid, fd));
+ CHK_ERRNO(libxl_domain_suspend(ctx, domid, fd, 0, NULL));
close(fd);
if (checkpoint)
@@ -3008,7 +3008,6 @@ static void migrate_domain(const char *domain_spec, const
char *rune,
pid_t child = -1;
int rc;
int send_fd = -1, recv_fd = -1;
- libxl_domain_suspend_info suspinfo;
char *away_domname;
char rc_buf;
uint8_t *config_data;
@@ -3030,9 +3029,7 @@ static void migrate_domain(const char *domain_spec, const
char *rune,
xtl_stdiostream_adjust_flags(logger, XTL_STDIOSTREAM_HIDE_PROGRESS, 0);
- memset(&suspinfo, 0, sizeof(suspinfo));
- suspinfo.flags |= XL_SUSPEND_LIVE;
- rc = libxl_domain_suspend(ctx, &suspinfo, domid, send_fd);
+ rc = libxl_domain_suspend(ctx, domid, send_fd, LIBXL_SUSPEND_LIVE, NULL);
if (rc) {
fprintf(stderr, "migration sender: libxl_domain_suspend
failed"
" (rc=%d)\n", rc);
@@ -6604,7 +6601,7 @@ int main_remus(int argc, char **argv)
}
/* Point of no return */
- rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd);
+ rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd,
0);
/* If we are here, it means backup has failed/domain suspend failed.
* Try to resume the domain and exit gracefully.
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 05/19] libxl: domain save/restore: run in a separate process
libxenctrl expects to be able to simply run the save or restore
operation synchronously. This won''t work well in a process which is
trying to handle multiple domains.
The options are:
- Block such a whole process (eg, the whole of libvirt) while
migration completes (or until it fails).
- Create a thread to run xc_domain_save and xc_domain_restore on.
This is quite unpalatable. Multithreaded programming is error
prone enough without generating threads in libraries, particularly
if the thread does some very complex operation.
- Fork and run the operation in the child without execing. This is
no good because we would need to negotiate with the caller about
fds we would inherit (and we might be a very large process).
- Fork and exec a helper.
Of these options the latter is the most palatable.
Consequently:
* A new helper program libxl-save-helper (which does both save and
restore). It will be installed in /usr/lib/xen/bin. It does not
link against libxl, only libxc, and its error handling does not
need to be very advanced. It does contain a plumbing through of
the logging interface into the callback stream.
* A small ad-hoc protocol between the helper and libxl which allows
log messages and the libxc callbacks to be passed up and down.
Protocol doc comment is in libxl_save_helper.c.
* To avoid a lot of tedium the marshalling boilerplate (stubs for the
helper and the callback decoder for libxl) is generated with a
small perl script.
* Implement new functionality to spawn the helper, monitor its
output, provide responses, and check on its exit status.
* The functions libxl__xc_domain_restore_done and
libxl__xc_domain_save_done now turn out to want be called in the
same place. So make their state argument a void* so that the two
functions are type compatible.
The domain save path still writes the qemu savefile synchronously.
This will need to be fixed in a subsequent patch.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v3:
* Suppress errno value in debug message when helper reports successful
completion.
* Significant consequential changes to cope with changes to
earlier patches in the series.
Changes in v2:
* Helper path can be overridden by an environment variable for testing.
* Add a couple of debug logging messages re toolstack data.
* Fixes from testing.
* Helper protocol message lengths (and numbers) are 16-bit which
more clearly avoids piling lots of junk on the stack.
* Merged with remus changes.
* Callback implementations in libxl now called via pointers
so remus can have its own callbacks.
* Better namespace prefixes on autogenerated names etc.
* Autogenerator can generate debugging printfs too.
---
.gitignore | 1 +
.hgignore | 2 +
tools/libxl/Makefile | 21 ++-
tools/libxl/libxl_create.c | 22 ++-
tools/libxl/libxl_dom.c | 36 ++--
tools/libxl/libxl_internal.h | 56 +++++-
tools/libxl/libxl_save_callout.c | 332 +++++++++++++++++++++++++++++-
tools/libxl/libxl_save_helper.c | 279 +++++++++++++++++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 393 ++++++++++++++++++++++++++++++++++++
9 files changed, 1104 insertions(+), 38 deletions(-)
create mode 100644 tools/libxl/libxl_save_helper.c
create mode 100755 tools/libxl/libxl_save_msgs_gen.pl
diff --git a/.gitignore b/.gitignore
index 7770e54..3451e52 100644
--- a/.gitignore
+++ b/.gitignore
@@ -353,6 +353,7 @@ tools/libxl/_*.[ch]
tools/libxl/testidl
tools/libxl/testidl.c
tools/libxl/*.pyc
+tools/libxl/libxl-save-helper
tools/blktap2/control/tap-ctl
tools/firmware/etherboot/eb-roms.h
tools/firmware/etherboot/gpxe-git-snapshot.tar.gz
diff --git a/.hgignore b/.hgignore
index 27d8f79..05304ea 100644
--- a/.hgignore
+++ b/.hgignore
@@ -180,9 +180,11 @@
^tools/libxl/_.*\.c$
^tools/libxl/libxlu_cfg_y\.output$
^tools/libxl/xl$
+^tools/libxl/libxl-save-helper$
^tools/libxl/testidl$
^tools/libxl/testidl\.c$
^tools/libxl/tmp\..*$
+^tools/libxl/.*\.new$
^tools/libvchan/vchan-node[12]$
^tools/libaio/src/.*\.ol$
^tools/libaio/src/.*\.os$
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 1d8b80a..1b81963 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -67,25 +67,30 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o
libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o \
- libxl_save_callout.o \
+ libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
$(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
$(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include $(XEN_ROOT)/tools/config.h
-AUTOINCS= libxlu_cfg_y.h libxlu_cfg_l.h _libxl_list.h _paths.h
+AUTOINCS= libxlu_cfg_y.h libxlu_cfg_l.h _libxl_list.h _paths.h \
+ _libxl_save_msgs_callout.h _libxl_save_msgs_helper.h
AUTOSRCS= libxlu_cfg_y.c libxlu_cfg_l.c
+AUTOSRCS += _libxl_save_msgs_callout.c _libxl_save_msgs_helper.c
LIBXLU_OBJS = libxlu_cfg_y.o libxlu_cfg_l.o libxlu_cfg.o \
libxlu_disk_l.o libxlu_disk.o libxlu_vif.o libxlu_pci.o
$(LIBXLU_OBJS): CFLAGS += $(CFLAGS_libxenctrl) # For xentoollog.h
-CLIENTS = xl testidl
+CLIENTS = xl testidl libxl-save-helper
XL_OBJS = xl.o xl_cmdimpl.o xl_cmdtable.o xl_sxp.o
$(XL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) # For xentoollog.h
$(XL_OBJS): CFLAGS += $(CFLAGS_libxenlight)
$(XL_OBJS): CFLAGS += -include $(XEN_ROOT)/tools/config.h # libxl_json.h needs
it.
+SAVE_HELPER_OBJS = libxl_save_helper.o _libxl_save_msgs_helper.o
+$(SAVE_HELPER_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenlight)
+
testidl.o: CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenlight)
testidl.c: libxl_types.idl gentest.py libxl.h $(AUTOINCS)
$(PYTHON) gentest.py libxl_types.idl testidl.c.new
@@ -117,6 +122,12 @@ _libxl_list.h:
$(XEN_INCLUDE)/xen-external/bsd-sys-queue-h-seddery $(XEN_INCLUDE
perl $^ --prefix=libxl >$@.new
$(call move-if-changed,$@.new,$@)
+_libxl_save_msgs_helper.c _libxl_save_msgs_callout.c \
+_libxl_save_msgs_helper.h _libxl_save_msgs_callout.h: \
+ libxl_save_msgs_gen.pl
+ $(PERL) -w $< $@ >$@.new
+ $(call move-if-changed,$@.new,$@)
+
libxl.h: _libxl_types.h
libxl_json.h: _libxl_types_json.h
libxl_internal.h: _libxl_types_internal.h _paths.h
@@ -159,6 +170,9 @@ libxlutil.a: $(LIBXLU_OBJS)
xl: $(XL_OBJS) libxlutil.so libxenlight.so
$(CC) $(LDFLAGS) -o $@ $(XL_OBJS) libxlutil.so $(LDLIBS_libxenlight)
$(LDLIBS_libxenctrl) -lyajl $(APPEND_LDFLAGS)
+libxl-save-helper: $(SAVE_HELPER_OBJS) libxenlight.so
+ $(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS) $(LDLIBS_libxenctrl)
$(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
+
testidl: testidl.o libxlutil.so libxenlight.so
$(CC) $(LDFLAGS) -o $@ testidl.o libxlutil.so $(LDLIBS_libxenlight)
$(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
@@ -170,6 +184,7 @@ install: all
$(INSTALL_DIR) $(DESTDIR)$(BASH_COMPLETION_DIR)
$(INSTALL_DIR) $(DESTDIR)$(XEN_RUN_DIR)
$(INSTALL_PROG) xl $(DESTDIR)$(SBINDIR)
+ $(INSTALL_PROG) libxl-save-helper $(DESTDIR)$(PRIVATE_BINDIR)
$(INSTALL_PROG) libxenlight.so.$(MAJOR).$(MINOR) $(DESTDIR)$(LIBDIR)
ln -sf libxenlight.so.$(MAJOR).$(MINOR)
$(DESTDIR)$(LIBDIR)/libxenlight.so.$(MAJOR)
ln -sf libxenlight.so.$(MAJOR) $(DESTDIR)$(LIBDIR)/libxenlight.so
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 4439367..00705d8 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -601,7 +601,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
libxl_domain_build_info *const info = &d_config->b_info;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- struct restore_callbacks *const callbacks = &dcs->callbacks;
+ libxl__srm_restore_autogen_callbacks *const callbacks +
&dcs->shs.callbacks.restore.a;
if (rc) domcreate_rebuild_done(egc, dcs, rc);
@@ -641,7 +642,6 @@ static void domcreate_bootloader_done(libxl__egc *egc,
pae = libxl_defbool_val(info->u.hvm.pae);
no_incr_generationid =
!libxl_defbool_val(info->u.hvm.incr_generationid);
callbacks->toolstack_restore = libxl__toolstack_restore;
- callbacks->data = gc;
break;
case LIBXL_DOMAIN_TYPE_PV:
hvm = 0;
@@ -661,10 +661,24 @@ static void domcreate_bootloader_done(libxl__egc *egc,
libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0);
}
-void libxl__xc_domain_restore_done(libxl__egc *egc,
- libxl__domain_create_state *dcs,
+void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
+ unsigned long console_mfn, unsigned long genidad, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
+ STATE_AO_GC(dcs->ao);
+ libxl__domain_build_state *const state = &dcs->build_state;
+
+ state->store_mfn = store_mfn;
+ state->console_mfn = console_mfn;
+ state->vm_generationid_addr = genidad;
+ shs->need_results = 0;
+}
+
+void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int ret, int retval, int errnoval)
{
+ libxl__domain_create_state *dcs = dcs_void;
STATE_AO_GC(dcs->ao);
libxl_ctx *ctx = libxl__gc_owner(gc);
char **vments = NULL, **localents = NULL;
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index b48452c..fea0c94 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -465,16 +465,20 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t
domid,
}
int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
- uint32_t size, void *data)
+ uint32_t size, void *user)
{
- libxl__gc *gc = data;
- libxl_ctx *ctx = gc->owner;
+ libxl__save_helper_state *shs = user;
+ libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
+ STATE_AO_GC(dcs->ao);
+ libxl_ctx *ctx = CTX;
int i, ret;
const uint8_t *ptr = buf;
uint32_t count = 0, version = 0;
struct libxl__physmap_info* pi;
char *xs_path;
+ LOG(DEBUG,"domain=%"PRIu32" toolstack data
size=%"PRIu32, domid, size);
+
if (size < sizeof(version) + sizeof(count)) {
LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "wrong size");
return -1;
@@ -527,9 +531,10 @@ static void domain_suspend_done(libxl__egc *egc,
/*----- callbacks, called by xc_domain_save -----*/
int libxl__domain_suspend_common_switch_qemu_logdirty
- (int domid, unsigned int enable, void *data)
+ (int domid, unsigned enable, void *user)
{
- libxl__domain_suspend_state *dss = data;
+ libxl__save_helper_state *shs = user;
+ libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
STATE_AO_GC(dss->ao);
char *path;
bool rc;
@@ -595,9 +600,10 @@ int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid)
return 0;
}
-int libxl__domain_suspend_common_callback(void *data)
+int libxl__domain_suspend_common_callback(void *user)
{
- libxl__domain_suspend_state *dss = data;
+ libxl__save_helper_state *shs = user;
+ libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
STATE_AO_GC(dss->ao);
unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
int ret;
@@ -737,9 +743,9 @@ static inline char *save_helper(libxl__gc *gc, uint32_t
domid,
}
int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
- uint32_t *len, void *data)
+ uint32_t *len, void *dss_void)
{
- libxl__domain_suspend_state *dss = data;
+ libxl__domain_suspend_state *dss = dss_void;
STATE_AO_GC(dss->ao);
int i = 0;
char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL;
@@ -808,6 +814,8 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
ptr += sizeof(struct libxl__physmap_info) + namelen;
}
+ LOG(DEBUG,"domain=%"PRIu32" toolstack data
size=%"PRIu32, domid, *len);
+
return 0;
}
@@ -862,7 +870,8 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
const int live = dss->live;
const int debug = dss->debug;
const libxl_domain_remus_info *const r_info = dss->remus;
- struct save_callbacks *const callbacks = &dss->callbacks;
+ libxl__srm_save_autogen_callbacks *const callbacks +
&dss->shs.callbacks.save.a;
switch (type) {
case LIBXL_DOMAIN_TYPE_HVM: {
@@ -923,8 +932,7 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
callbacks->suspend = libxl__domain_suspend_common_callback;
callbacks->switch_qemu_logdirty =
libxl__domain_suspend_common_switch_qemu_logdirty;
- callbacks->toolstack_save = libxl__toolstack_save;
- callbacks->data = dss;
+ dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
libxl__xc_domain_save(egc, dss, vm_generationid_addr);
return;
@@ -933,10 +941,10 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
domain_suspend_done(egc, dss, rc);
}
-void libxl__xc_domain_save_done(libxl__egc *egc,
- libxl__domain_suspend_state *dss,
+void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
int rc, int retval, int errnoval)
{
+ libxl__domain_suspend_state *dss = dss_void;
STATE_AO_GC(dss->ao);
/* Convenience aliases */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e72b8d2..dbf7722 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -54,6 +54,7 @@
#include "libxl.h"
#include "_paths.h"
+#include "_libxl_save_msgs_callout.h"
#if __GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)
#define _hidden __attribute__((visibility("hidden")))
@@ -1773,6 +1774,51 @@ _hidden void
libxl__datacopier_kill(libxl__datacopier_state *dc);
_hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
+/*----- Save/restore helper (used by creation and suspend) -----*/
+
+typedef struct libxl__srm_save_callbacks {
+ libxl__srm_save_autogen_callbacks a;
+ int (*toolstack_save)(uint32_t domid, uint8_t **buf,
+ uint32_t *len, void *data);
+} libxl__srm_save_callbacks;
+
+typedef struct libxl__srm_restore_callbacks {
+ libxl__srm_restore_autogen_callbacks a;
+} libxl__srm_restore_callbacks;
+
+/* a pointer to this struct is also passed as "user" to the
+ * save callout helper callback functions */
+typedef struct libxl__save_helper_state {
+ /* public, caller of run_helper initialises */
+ libxl__ao *ao;
+ uint32_t domid;
+ union {
+ libxl__srm_save_callbacks save;
+ libxl__srm_restore_callbacks restore;
+ } callbacks;
+ int (*recv_callback)(const unsigned char *msg, uint32_t len, void *user);
+ void (*completion_callback)(libxl__egc *egc, void *caller_state,
+ int rc, int retval, int errnoval);
+ void *caller_state;
+ int need_results; /* set to 0 or 1 by caller of run_helper;
+ * if set to 1 then the ultimate caller''s
+ * results function must set it to 0 */
+ /* private */
+ int rc;
+ int completed; /* retval/errnoval valid iff completed */
+ int retval, errnoval; /* from xc_domain_save / xc_domain_restore */
+ libxl__carefd *pipes[2]; /* 0 = helper''s stdin, 1 =
helper''s stdout */
+ libxl__ev_fd readable;
+ libxl__ev_child child;
+ const char *stdin_what, *stdout_what;
+ FILE *toolstack_data_file;
+
+ libxl__egc *egc; /* valid only for duration of each event callback;
+ * is here in this struct for the benefit of the
+ * marshalling and xc callback functions */
+} libxl__save_helper_state;
+
+
/*----- Domain suspend (save) state structure -----*/
typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
@@ -1798,7 +1844,7 @@ struct libxl__domain_suspend_state {
int xcflags;
int guest_responded;
int interval; /* checkpoint interval (for Remus) */
- struct save_callbacks callbacks;
+ libxl__save_helper_state shs;
};
@@ -1910,7 +1956,7 @@ struct libxl__domain_create_state {
libxl__stub_dm_spawn_state dmss;
/* If we''re not doing stubdom, we use only dmss.dm,
* for the non-stubdom device model. */
- struct restore_callbacks callbacks;
+ libxl__save_helper_state shs;
};
/*----- Domain suspend (save) functions -----*/
@@ -1926,8 +1972,7 @@ _hidden void libxl__xc_domain_save(libxl__egc*,
libxl__domain_suspend_state*,
/* If rc==0 then retval is the return value from xc_domain_save
* and errnoval is the errno value it provided.
* If rc!=0, retval and errnoval are undefined. */
-_hidden void libxl__xc_domain_save_done(libxl__egc*,
- libxl__domain_suspend_state*,
+_hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
int rc, int retval, int errnoval);
_hidden int libxl__domain_suspend_common_callback(void *data);
@@ -1945,8 +1990,7 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc,
/* If rc==0 then retval is the return value from xc_domain_save
* and errnoval is the errno value it provided.
* If rc!=0, retval and errnoval are undefined. */
-_hidden void libxl__xc_domain_restore_done(libxl__egc *egc,
- libxl__domain_create_state *dcs,
+_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int rc, int retval, int errnoval);
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 1b481ab..a39f434 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -16,6 +16,19 @@
#include "libxl_internal.h"
+static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
+ const char *mode_arg, int data_fd,
+ const unsigned long *argnums, int num_argnums);
+
+static void helper_failed(libxl__egc*, libxl__save_helper_state *shs, int rc);
+static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
+ int fd, short events, short revents);
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status);
+static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs);
+
+/*----- entrypoints -----*/
+
void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
int hvm, int pae, int superpages,
int no_incr_generationid)
@@ -27,22 +40,319 @@ void libxl__xc_domain_restore(libxl__egc *egc,
libxl__domain_create_state *dcs,
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- int r = xc_domain_restore(CTX->xch, restore_fd, domid,
- state->store_port, &state->store_mfn,
- state->store_domid, state->console_port,
- &state->console_mfn,
state->console_domid,
- hvm, pae, superpages, no_incr_generationid,
- &state->vm_generationid_addr,
&dcs->callbacks);
- libxl__xc_domain_restore_done(egc, dcs, 0, r, errno);
+ unsigned cbflags = libxl__srm_callout_enumcallbacks_restore
+ (&dcs->shs.callbacks.restore.a);
+
+ const unsigned long argnums[] = {
+ restore_fd, domid,
+ state->store_port,
+ state->store_domid, state->console_port,
+ state->console_domid,
+ hvm, pae, superpages, no_incr_generationid,
+ cbflags,
+ };
+
+ dcs->shs.ao = ao;
+ dcs->shs.domid = domid;
+ dcs->shs.recv_callback = libxl__srm_callout_received_restore;
+ dcs->shs.completion_callback = libxl__xc_domain_restore_done;
+ dcs->shs.caller_state = dcs;
+ dcs->shs.need_results = 1;
+ dcs->shs.toolstack_data_file = 0;
+
+ run_helper(egc, &dcs->shs, "--restore-domain", restore_fd,
+ argnums, ARRAY_SIZE(argnums));
}
void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
unsigned long vm_generationid_addr)
{
STATE_AO_GC(dss->ao);
- int r;
+ int r, rc;
+ uint32_t toolstack_data_len;
+
+ /* Resources we need to free */
+ uint8_t *toolstack_data_buf = 0;
+
+ unsigned cbflags = libxl__srm_callout_enumcallbacks_save
+ (&dss->shs.callbacks.save.a);
+
+ r = libxl__toolstack_save(dss->domid, &toolstack_data_buf,
+ &toolstack_data_len, dss);
+ if (r) { rc = ERROR_FAIL; goto out; }
+
+ dss->shs.toolstack_data_file = tmpfile();
+ if (!dss->shs.toolstack_data_file) {
+ LOGE(ERROR, "cannot create toolstack data tmpfile");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ int toolstack_data_fd = fileno(dss->shs.toolstack_data_file);
+
+ r = libxl_write_exactly(CTX, toolstack_data_fd,
+ toolstack_data_buf, toolstack_data_len,
+ "toolstack data tmpfile", 0);
+ if (r) { rc = ERROR_FAIL; goto out; }
+
+ r = lseek(toolstack_data_fd, 0, SEEK_SET);
+ if (r) {
+ LOGE(ERROR, "rewind toolstack data tmpfile");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ const unsigned long argnums[] = {
+ dss->fd, dss->domid, 0, 0, dss->xcflags, dss->hvm,
vm_generationid_addr,
+ toolstack_data_fd, toolstack_data_len,
+ cbflags,
+ };
+
+ dss->shs.ao = ao;
+ dss->shs.domid = dss->domid;
+ dss->shs.recv_callback = libxl__srm_callout_received_save;
+ dss->shs.completion_callback = libxl__xc_domain_save_done;
+ dss->shs.caller_state = dss;
+ dss->shs.need_results = 0;
+
+ free(toolstack_data_buf);
+
+ run_helper(egc, &dss->shs, "--save-domain", dss->fd,
+ argnums, ARRAY_SIZE(argnums));
+ return;
+
+ out:
+ free(toolstack_data_buf);
+ if (dss->shs.toolstack_data_file)
fclose(dss->shs.toolstack_data_file);
+
+ libxl__xc_domain_save_done(egc, dss, rc, 0, 0);
+}
+
+
+/*----- helper execution -----*/
+
+static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
+ const char *mode_arg, int data_fd,
+ const unsigned long *argnums, int num_argnums)
+{
+ STATE_AO_GC(shs->ao);
+ const char *args[3 + num_argnums];
+ const char **arg = args;
+ int i, rc;
+
+ /* Resources we must free */
+ libxl__carefd *childs_pipes[2] = { 0,0 };
+
+ /* Convenience aliases */
+ const uint32_t domid = shs->domid;
+
+ shs->rc = 0;
+ shs->completed = 0;
+ shs->pipes[0] = shs->pipes[1] = 0;
+ libxl__ev_fd_init(&shs->readable);
+ libxl__ev_child_init(&shs->child);
+
+ shs->stdin_what = GCSPRINTF("domain %"PRIu32"
save/restore helper"
+ " stdin pipe", domid);
+ shs->stdout_what = GCSPRINTF("domain %"PRIu32"
save/restore helper"
+ " stdout pipe", domid);
+
+ *arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC "/"
"libxl-save-helper";
+ *arg++ = mode_arg;
+ for (i=0; i<num_argnums; i++)
+ *arg++ = GCSPRINTF("%lu", argnums[i]);
+ *arg++ = 0;
+ assert(arg == args + ARRAY_SIZE(args));
+
+ libxl__carefd_begin();
+ for (i=0; i<2; i++) {
+ int fds[2];
+ if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; }
+ childs_pipes[i] = libxl__carefd_record(CTX, fds[i]);
+ shs->pipes[i] = libxl__carefd_record(CTX, fds[!i]);
+ }
+ libxl__carefd_unlock();
+
+ pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited);
+ if (!pid) {
+ libxl_fd_set_cloexec(CTX, data_fd, 0);
+ libxl__exec(gc,
+ libxl__carefd_fd(childs_pipes[0]),
+ libxl__carefd_fd(childs_pipes[1]),
+ -1,
+ args[0], (char**)args, 0);
+ }
+
+ libxl__carefd_close(childs_pipes[0]);
+ libxl__carefd_close(childs_pipes[1]);
+
+ rc = libxl__ev_fd_register(gc, &shs->readable,
helper_stdout_readable,
+ libxl__carefd_fd(shs->pipes[1]),
POLLIN|POLLPRI);
+ if (rc) goto out;
+ return;
+
+ out:
+ libxl__carefd_close(childs_pipes[0]);
+ libxl__carefd_close(childs_pipes[1]);
+ helper_failed(egc, shs, rc);;
+}
+
+static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
+ int rc)
+{
+ STATE_AO_GC(shs->ao);
+
+ if (!shs->rc)
+ shs->rc = rc;
+
+ libxl__ev_fd_deregister(gc, &shs->readable);
+
+ if (!libxl__ev_child_inuse(&shs->child)) {
+ helper_done(egc, shs);
+ return;
+ }
+
+ int r = kill(shs->child.pid, SIGKILL);
+ if (r) LOGE(WARN, "failed to kill save/restore helper [%lu]",
+ (unsigned long)shs->child.pid);
+}
+
+static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
+ int fd, short events, short revents)
+{
+ libxl__save_helper_state *shs = CONTAINER_OF(ev, *shs, readable);
+ STATE_AO_GC(shs->ao);
+ int rc, errnoval;
+
+ if (revents & POLLPRI) {
+ LOG(ERROR, "%s signaled POLLERR", shs->stdout_what);
+ rc = ERROR_FAIL;
+ out:
+ /* this is here because otherwise we bypass the decl of msg[] */
+ helper_failed(egc, shs, rc);
+ return;
+ }
+
+ uint16_t msglen;
+ errnoval = libxl_read_exactly(CTX, fd, &msglen, sizeof(msglen),
+ shs->stdout_what, "ipc msg
header");
+ if (errnoval) { rc = ERROR_FAIL; goto out; }
+
+ unsigned char msg[msglen];
+ errnoval = libxl_read_exactly(CTX, fd, msg, msglen,
+ shs->stdout_what, "ipc msg
body");
+ if (errnoval) { rc = ERROR_FAIL; goto out; }
+
+ shs->egc = egc;
+ shs->recv_callback(msg, msglen, shs);
+ shs->egc = 0;
+ return;
+}
+
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status)
+{
+ libxl__save_helper_state *shs = CONTAINER_OF(ch, *shs, child);
+ STATE_AO_GC(shs->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = shs->domid;
+
+ const char *what + GCSPRINTF("domain %"PRIu32"
save/restore helper", domid);
+
+ if (status) {
+ libxl_report_child_exitstatus(CTX, XTL_ERROR, what, pid, status);
+ shs->rc = ERROR_FAIL;
+ }
+
+ if (shs->need_results) {
+ if (!shs->rc)
+ LOG(ERROR,"%s exited without providing results",what);
+ shs->rc = ERROR_FAIL;
+ }
+
+ if (!shs->completed) {
+ if (!shs->rc)
+ LOG(ERROR,"%s exited without signaling completion",what);
+ shs->rc = ERROR_FAIL;
+ }
+
+ helper_done(egc, shs);
+ return;
+}
+
+static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs)
+{
+ STATE_AO_GC(shs->ao);
+
+ libxl__ev_fd_deregister(gc, &shs->readable);
+ libxl__carefd_close(shs->pipes[0]); shs->pipes[0] = 0;
+ libxl__carefd_close(shs->pipes[1]); shs->pipes[1] = 0;
+ assert(!libxl__ev_child_inuse(&shs->child));
+ if (shs->toolstack_data_file) fclose(shs->toolstack_data_file);
+
+ shs->egc = egc;
+ shs->completion_callback(egc, shs->caller_state,
+ shs->rc, shs->retval, shs->errnoval);
+ shs->egc = 0;
+}
+
+/*----- generic helpers for the autogenerated code -----*/
+
+const libxl__srm_save_autogen_callbacks*
+libxl__srm_callout_get_callbacks_save(void *user)
+{
+ libxl__save_helper_state *shs = user;
+ return &shs->callbacks.save.a;
+}
+
+const libxl__srm_restore_autogen_callbacks*
+libxl__srm_callout_get_callbacks_restore(void *user)
+{
+ libxl__save_helper_state *shs = user;
+ return &shs->callbacks.restore.a;
+}
+
+void libxl__srm_callout_sendreply(int r, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ libxl__egc *egc = shs->egc;
+ STATE_AO_GC(shs->ao);
+ int errnoval;
+
+ errnoval = libxl_write_exactly(CTX, libxl__carefd_fd(shs->pipes[0]),
+ &r, sizeof(r), shs->stdin_what,
+ "callback return value");
+ if (errnoval)
+ helper_failed(egc, shs, ERROR_FAIL);
+}
+
+void libxl__srm_callout_callback_log(uint32_t level, uint32_t errnoval,
+ const char *context, const char *formatted, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ STATE_AO_GC(shs->ao);
+ xtl_log(CTX->lg, level, errnoval, context, "%s", formatted);
+}
+
+void libxl__srm_callout_callback_progress(const char *context,
+ const char *doing_what, unsigned long done,
+ unsigned long total, void *user)
+{
+ libxl__save_helper_state *shs = user;
+ STATE_AO_GC(shs->ao);
+ xtl_progress(CTX->lg, context, doing_what, done, total);
+}
+
+int libxl__srm_callout_callback_complete(int retval, int errnoval,
+ void *user)
+{
+ libxl__save_helper_state *shs = user;
+ STATE_AO_GC(shs->ao);
- r = xc_domain_save(CTX->xch, dss->fd, dss->domid, 0, 0,
dss->xcflags,
- &dss->callbacks, dss->hvm,
vm_generationid_addr);
- libxl__xc_domain_save_done(egc, dss, 0, r, errno);
+ shs->completed = 1;
+ shs->retval = retval;
+ shs->errnoval = errnoval;
+ libxl__ev_fd_deregister(gc, &shs->readable);
+ return 0;
}
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
new file mode 100644
index 0000000..d29f1f7
--- /dev/null
+++ b/tools/libxl/libxl_save_helper.c
@@ -0,0 +1,279 @@
+/*
+ * Copyright (C) 2012 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+/*
+ * The libxl-save-helper utility speaks a protocol to its caller for
+ * the callbacks. The protocol is as follows.
+ *
+ * The helper talks on stdin and stdout, in binary in machine
+ * endianness. The helper speaks first, and only when it has a
+ * callback to make. It writes a 16-bit number being the message
+ * length, and then the message body.
+ *
+ * Each message starts with a 16-bit number indicating which of the
+ * messages it is, and then some arguments in a binary marshalled form.
+ * If the callback does not need a reply (it returns void), the helper
+ * just continues. Otherwise the helper waits for its caller to send a
+ * single int which is to be the return value from the callback.
+ *
+ * Where feasible the stubs and callbacks have prototypes identical to
+ * those required by xc_domain_save and xc_domain_restore, so that the
+ * autogenerated functions can be used/provided directly.
+ *
+ * The actual messages are in the array @msgs in libxl_save_msgs_gen.pl
+ */
+
+#include "libxl_osdeps.h"
+
+#include <stdlib.h>
+#include <unistd.h>
+#include <assert.h>
+#include <inttypes.h>
+
+#include "libxl.h"
+
+#include "xenctrl.h"
+#include "xenguest.h"
+#include "_libxl_save_msgs_helper.h"
+
+/*----- globals -----*/
+
+static const char *program = "libxl-save-helper";
+static xentoollog_logger *logger;
+static xc_interface *xch;
+
+/*----- error handling -----*/
+
+static void fail(int errnoval, const char *fmt, ...)
+ __attribute__((noreturn,format(printf,2,3)));
+static void fail(int errnoval, const char *fmt, ...)
+{
+ va_list al;
+ va_start(al,fmt);
+ xtl_logv(logger,XTL_ERROR,errnoval,program,fmt,al);
+ exit(-1);
+}
+
+static int read_exactly(int fd, void *buf, size_t len)
+/* returns 0 if we get eof, even if we got it midway through; 1 if ok */
+{
+ while (len) {
+ ssize_t r = read(fd, buf, len);
+ if (r<=0) return r;
+ assert(r <= len);
+ len -= r;
+ buf = (char*)buf + r;
+ }
+ return 1;
+}
+
+static void *xmalloc(size_t sz)
+{
+ if (!sz) return 0;
+ void *r = malloc(sz);
+ if (!r) { perror("memory allocation failed"); exit(-1); }
+ return r;
+}
+
+/*----- logger -----*/
+
+typedef struct {
+ xentoollog_logger vtable;
+} xentoollog_logger_tellparent;
+
+static void tellparent_vmessage(xentoollog_logger *logger_in,
+ xentoollog_level level,
+ int errnoval,
+ const char *context,
+ const char *format,
+ va_list al)
+{
+ char *formatted;
+ int r = vasprintf(&formatted, format, al);
+ if (r < 0) { perror("memory allocation failed during
logging"); exit(-1); }
+ helper_stub_log(level, errnoval, context, formatted, 0);
+ free(formatted);
+}
+
+static void tellparent_progress(struct xentoollog_logger *logger_in,
+ const char *context,
+ const char *doing_what, int percent,
+ unsigned long done, unsigned long total)
+{
+ helper_stub_progress(context, doing_what, done, total, 0);
+}
+
+static void tellparent_destroy(struct xentoollog_logger *logger_in)
+{
+ abort();
+}
+
+static xentoollog_logger_tellparent *createlogger_tellparent(void)
+{
+ xentoollog_logger_tellparent newlogger;
+ return XTL_NEW_LOGGER(tellparent, newlogger);
+}
+
+/*----- helper functions called by autogenerated stubs -----*/
+
+unsigned char * helper_allocbuf(int len, void *user)
+{
+ return xmalloc(len);
+}
+
+static void transmit(const unsigned char *msg, int len, void *user)
+{
+ while (len) {
+ int r = write(1, msg, len);
+ if (r<0) { perror("write"); exit(-1); }
+ assert(r >= 0);
+ assert(r <= len);
+ len -= r;
+ msg += r;
+ }
+}
+
+void helper_transmitmsg(unsigned char *msg_freed, int len_in, void *user)
+{
+ assert(len_in < 64*1024);
+ uint16_t len = len_in;
+ transmit((const void*)&len, sizeof(len), user);
+ transmit(msg_freed, len, user);
+ free(msg_freed);
+}
+
+int helper_getreply(void *user)
+{
+ int v;
+ int r = read_exactly(0, &v, sizeof(v));
+ if (r<=0) exit(-2);
+ return v;
+}
+
+/*----- other callbacks -----*/
+
+static int toolstack_save_fd;
+static uint32_t toolstack_save_len;
+
+static int toolstack_save_cb(uint32_t domid, uint8_t **buf,
+ uint32_t *len, void *data)
+{
+ assert(toolstack_save_fd > 0);
+
+ *buf = xmalloc(toolstack_save_len);
+ int r = read_exactly(toolstack_save_fd, *buf, toolstack_save_len);
+ if (r<0) fail(errno,"read toolstack data");
+ if (r==0) fail(0,"read toolstack data eof");
+
+ toolstack_save_fd = -1;
+ *len = toolstack_save_len;
+ return 0;
+}
+
+static void startup(const char *op) {
+ logger = (xentoollog_logger*)createlogger_tellparent();
+ if (!logger) {
+ fprintf(stderr, "%s: cannot initialise logger\n", program);
+ exit(-1);
+ }
+
+ xtl_log(logger,XTL_DEBUG,0,program,"starting %s",op);
+
+ xch = xc_interface_open(logger,logger,0);
+ if (!xch) fail(errno,"xc_interface_open failed");
+}
+
+static void complete(int retval) {
+ int errnoval = retval ? errno : 0; /* suppress irrelevant errnos */
+ xtl_log(logger,XTL_DEBUG,errnoval,program,"complete
r=%d",retval);
+ helper_stub_complete(retval,errnoval,0);
+ exit(0);
+}
+
+static struct save_callbacks helper_save_callbacks;
+static struct restore_callbacks helper_restore_callbacks;
+
+int main(int argc, char **argv)
+{
+ int r;
+
+#define NEXTARG (++argv, assert(*argv), *argv)
+
+ const char *mode = *++argv;
+ assert(mode);
+
+ if (!strcmp(mode,"--save-domain")) {
+
+ int io_fd = atoi(NEXTARG);
+ uint32_t dom = strtoul(NEXTARG,0,10);
+ uint32_t max_iters = strtoul(NEXTARG,0,10);
+ uint32_t max_factor = strtoul(NEXTARG,0,10);
+ uint32_t flags = strtoul(NEXTARG,0,10);
+ int hvm = atoi(NEXTARG);
+ unsigned long genidad = strtoul(NEXTARG,0,10);
+ toolstack_save_fd = atoi(NEXTARG);
+ toolstack_save_len = strtoul(NEXTARG,0,10);
+ unsigned cbflags = strtoul(NEXTARG,0,10);
+ assert(!*++argv);
+
+ helper_save_callbacks.toolstack_save = toolstack_save_cb;
+ helper_setcallbacks_save(&helper_save_callbacks, cbflags);
+
+ startup("save");
+ r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
+ &helper_save_callbacks, hvm, genidad);
+ complete(r);
+
+ } else if (!strcmp(mode,"--restore-domain")) {
+
+ int io_fd = atoi(NEXTARG);
+ uint32_t dom = strtoul(NEXTARG,0,10);
+ unsigned store_evtchn = strtoul(NEXTARG,0,10);
+ domid_t store_domid = strtoul(NEXTARG,0,10);
+ unsigned console_evtchn = strtoul(NEXTARG,0,10);
+ domid_t console_domid = strtoul(NEXTARG,0,10);
+ unsigned int hvm = strtoul(NEXTARG,0,10);
+ unsigned int pae = strtoul(NEXTARG,0,10);
+ int superpages = strtoul(NEXTARG,0,10);
+ int no_incr_genidad = strtoul(NEXTARG,0,10);
+ unsigned cbflags = strtoul(NEXTARG,0,10);
+ assert(!*++argv);
+
+ helper_setcallbacks_restore(&helper_restore_callbacks, cbflags);
+
+ unsigned long store_mfn = 0;
+ unsigned long console_mfn = 0;
+ unsigned long genidad = 0;
+
+ startup("restore");
+ r = xc_domain_restore(xch, io_fd, dom, store_evtchn, &store_mfn,
+ store_domid, console_evtchn, &console_mfn,
+ console_domid, hvm, pae, superpages,
+ no_incr_genidad, &genidad,
+ &helper_restore_callbacks);
+ helper_stub_restore_results(store_mfn,console_mfn,genidad,0);
+ complete(r);
+
+ } else {
+ assert(!"unexpected mode argument");
+ }
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_save_msgs_gen.pl
b/tools/libxl/libxl_save_msgs_gen.pl
new file mode 100755
index 0000000..cd0837e
--- /dev/null
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -0,0 +1,393 @@
+#!/usr/bin/perl -w
+
+use warnings;
+use strict;
+use POSIX;
+
+our $debug = 0; # produce copius debugging output at run-time?
+
+our @msgs = (
+ # flags:
+ # s - applicable to save
+ # r - applicable to restore
+ # c - function pointer in callbacks struct rather than fixed function
+ # x - function pointer is in struct {save,restore}_callbacks
+ # and its null-ness needs to be passed through to the
helper''s xc
+ # W - needs a return value; callback is synchronous
+ [ 1, ''sr'', "log",
[qw(uint32_t level
+ uint32_t errnoval
+ STRING context
+ STRING formatted)] ],
+ [ 2, ''sr'', "progress",
[qw(STRING context
+ STRING doing_what),
+ ''unsigned
long'', ''done'',
+ ''unsigned
long'', ''total''] ],
+ [ 3, ''scxW'', "suspend", [] ],
+ [ 4, ''scxW'', "postcopy", [] ],
+ [ 5, ''scxW'', "checkpoint", [] ],
+ [ 6, ''scxW'', "switch_qemu_logdirty",
[qw(int domid
+ unsigned enable)] ],
+ # toolstack_save done entirely `by hand''
+ [ 7, ''rcxW'', "toolstack_restore",
[qw(uint32_t domid
+ BLOCK tsdata)] ],
+ [ 8, ''r'', "restore_results",
[''unsigned long'', ''store_mfn'',
+ ''unsigned
long'', ''console_mfn'',
+ ''unsigned
long'', ''genidad''] ],
+ [ 9, ''srW'', "complete",
[qw(int retval
+ int errnoval)] ],
+);
+
+#----------------------------------------
+
+our %cbs;
+our %func;
+our %func_ah;
+our @outfuncs;
+our %out_decls;
+our %out_body;
+our %msgnum_used;
+
+die unless @ARGV==1;
+die if $ARGV[0] =~ m/^-/;
+
+our ($intendedout) = @ARGV;
+
+$intendedout =~ m/([a-z]+)\.([ch])$/ or die;
+my ($want_ah, $ch) = ($1, $2);
+
+my $declprefix = '''';
+
+foreach my $ah (qw(callout helper)) {
+ $out_body{$ah} .= <<END.($ah eq ''callout'' ?
<<END : <<END);
+#include "libxl_osdeps.h"
+
+#include <assert.h>
+#include <string.h>
+#include <stdint.h>
+#include <limits.h>
+END
+
+#include "libxl_internal.h"
+
+END
+
+#include "_libxl_save_msgs_${ah}.h"
+#include <xenctrl.h>
+#include <xenguest.h>
+
+END
+}
+
+die $want_ah unless defined $out_body{$want_ah};
+
+sub f_decl ($$$$) {
+ my ($name, $ah, $c_rtype, $c_decl) = @_;
+ $out_decls{$name} = "${declprefix}$c_rtype $name$c_decl;\n";
+ $func{$name} = "$c_rtype $name$c_decl\n{\n" . ($func{$name} ||
'''');
+ $func_ah{$name} = $ah;
+}
+
+sub f_more ($$) {
+ my ($name, $addbody) = @_;
+ $func{$name} ||= '''';
+ $func{$name} .= $addbody;
+ push @outfuncs, $name;
+}
+
+our $libxl = "libxl__srm";
+our $callback = "${libxl}_callout_callback";
+our $receiveds = "${libxl}_callout_received";
+our $sendreply = "${libxl}_callout_sendreply";
+our $getcallbacks = "${libxl}_callout_get_callbacks";
+our $enumcallbacks = "${libxl}_callout_enumcallbacks";
+sub cbtype ($) { "${libxl}_".$_[0]."_autogen_callbacks"; };
+
+f_decl($sendreply, ''callout'', ''void'',
"(int r, void *user)");
+
+our $helper = "helper";
+our $encode = "${helper}_stub";
+our $allocbuf = "${helper}_allocbuf";
+our $transmit = "${helper}_transmitmsg";
+our $getreply = "${helper}_getreply";
+our $setcallbacks = "${helper}_setcallbacks";
+
+f_decl($allocbuf, ''helper'', ''unsigned char
*'', ''(int len, void *user)'');
+f_decl($transmit, ''helper'', ''void'',
+ ''(unsigned char *msg_freed, int len, void *user)'');
+f_decl($getreply, ''helper'', ''int'',
''(void *user)'');
+
+sub typeid ($) { my ($t) = @_; $t =~ s/\W/_/; return $t; };
+
+$out_body{''callout''} .= <<END;
+static int bytes_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ void *result, int rlen)
+{
+ if (endmsg - *msg < rlen) return 0;
+ memcpy(result,*msg,rlen);
+ *msg += rlen;
+ return 1;
+}
+
+END
+$out_body{''helper''} .= <<END;
+static void bytes_put(unsigned char *const buf, int *len,
+ const void *value, int vlen)
+{
+ assert(vlen < INT_MAX/2 - *len);
+ if (buf)
+ memcpy(buf + *len, value, vlen);
+ *len += vlen;
+}
+
+END
+
+foreach my $simpletype (qw(int uint16_t uint32_t unsigned), ''unsigned
long'') {
+ my $typeid = typeid($simpletype);
+ $out_body{''callout''} .= <<END;
+static int ${typeid}_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ $simpletype *result)
+{
+ return bytes_get(msg, endmsg, result, sizeof(*result));
+}
+
+END
+ $out_body{''helper''} .= <<END;
+static void ${typeid}_put(unsigned char *const buf, int *len,
+ const $simpletype value)
+{
+ bytes_put(buf, len, &value, sizeof(value));
+}
+
+END
+}
+
+$out_body{''callout''} .= <<END;
+static int BLOCK_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ const uint8_t **result, uint32_t *result_size)
+{
+ if (!uint32_t_get(msg,endmsg,result_size)) return 0;
+ if (endmsg - *msg < *result_size) return 0;
+ *result = (const void*)*msg;
+ *msg += *result_size;
+ return 1;
+}
+
+static int STRING_get(const unsigned char **msg,
+ const unsigned char *const endmsg,
+ const char **result)
+{
+ const uint8_t *data;
+ uint32_t datalen;
+ if (!BLOCK_get(msg,endmsg,&data,&datalen)) return 0;
+ if (datalen == 0) return 0;
+ if (data[datalen-1] != ''\\0'') return 0;
+ *result = (const void*)data;
+ return 1;
+}
+
+END
+$out_body{''helper''} .= <<END;
+static void BLOCK_put(unsigned char *const buf,
+ int *len,
+ const uint8_t *bytes, uint32_t size)
+{
+ uint32_t_put(buf, len, size);
+ bytes_put(buf, len, bytes, size);
+}
+
+static void STRING_put(unsigned char *const buf,
+ int *len,
+ const char *string)
+{
+ size_t slen = strlen(string);
+ assert(slen < INT_MAX / 4);
+ assert(slen < (uint32_t)0x40000000);
+ BLOCK_put(buf, len, (const void*)string, slen+1);
+}
+
+END
+
+foreach my $sr (qw(save restore)) {
+ f_decl("${getcallbacks}_${sr}", ''callout'',
+ "const ".cbtype($sr)." *",
+ "(void *data)");
+
+ f_decl("${receiveds}_${sr}", ''callout'',
''int'',
+ "(const unsigned char *msg, uint32_t len, void *user)");
+
+ f_decl("${enumcallbacks}_${sr}", ''callout'',
''unsigned'',
+ "(const ".cbtype($sr)." *cbs)");
+ f_more("${enumcallbacks}_${sr}", " unsigned cbflags =
0;\n");
+
+ f_decl("${setcallbacks}_${sr}", ''helper'',
''void'',
+ "(struct ${sr}_callbacks *cbs, unsigned cbflags)");
+
+ f_more("${receiveds}_${sr}", <<END.($debug ? <<END :
'''').<<END);
+ const unsigned char *const endmsg = msg + len;
+ uint16_t mtype;
+ if (!uint16_t_get(&msg,endmsg,&mtype)) return 0;
+END
+ fprintf(stderr,"libxl callout receiver: got len=%u
mtype=%u\\n",len,mtype);
+END
+ switch (mtype) {
+
+END
+
+ $cbs{$sr} = "typedef struct ".cbtype($sr)." {\n";
+}
+
+foreach my $msginfo (@msgs) {
+ my ($msgnum, $flags, $name, $args) = @$msginfo;
+ die if $msgnum_used{$msgnum}++;
+
+ my $f_more_sr = sub {
+ my ($contents_spec, $fnamebase) = @_;
+ $fnamebase ||= "${receiveds}";
+ foreach my $sr (qw(save restore)) {
+ $sr =~ m/^./;
+ next unless $flags =~ m/$&/;
+ my $contents = (!ref $contents_spec) ? $contents_spec :
+ $contents_spec->($sr);
+ f_more("${fnamebase}_${sr}", $contents);
+ }
+ };
+
+ $f_more_sr->(" case $msgnum: { /* $name */\n");
+ if ($flags =~ m/W/) {
+ $f_more_sr->(" int r;\n");
+ }
+
+ my $c_rtype_helper = $flags =~ m/W/ ? ''int'' :
''void'';
+ my $c_rtype_callout = $flags =~ m/W/ ? ''int'' :
''void'';
+ my $c_decl = ''('';
+ my $c_callback_args = '''';
+
+ f_more("${encode}_$name", <<END.($debug ? <<END :
'''').<<END);
+ unsigned char *buf = 0;
+ int len = 0, allocd = 0;
+
+END
+ fprintf(stderr,"libxl-save-helper: encoding $name\\n");
+END
+ for (;;) {
+ uint16_t_put(buf, &len, $msgnum /* $name */);
+END
+
+ my @args = @$args;
+ my $c_recv = '''';
+ my ($argtype, $arg);
+ while (($argtype, $arg, @args) = @args) {
+ my $typeid = typeid($argtype);
+ my $c_args = "$arg";
+ my $c_get_args = "&$arg";
+ if ($argtype eq ''STRING'') {
+ $c_decl .= "const char *$arg, ";
+ $f_more_sr->(" const char *$arg;\n");
+ } elsif ($argtype eq ''BLOCK'') {
+ $c_decl .= "const uint8_t *$arg, uint32_t ${arg}_size, ";
+ $c_args .= ", ${arg}_size";
+ $c_get_args .= ",&${arg}_size";
+ $f_more_sr->(" const uint8_t *$arg;\n".
+ " uint32_t ${arg}_size;\n");
+ } else {
+ $c_decl .= "$argtype $arg, ";
+ $f_more_sr->(" $argtype $arg;\n");
+ }
+ $c_callback_args .= "$c_args, ";
+ $c_recv.+ " if
(!${typeid}_get(&msg,endmsg,$c_get_args)) return 0;\n";
+ f_more("${encode}_$name", " ${typeid}_put(buf, &len,
$c_args);\n");
+ }
+ $f_more_sr->($c_recv);
+ $c_decl .= "void *user)";
+ $c_callback_args .= "user";
+
+ $f_more_sr->(" if (msg != endmsg) return 0;\n");
+
+ my $c_callback;
+ if ($flags !~ m/c/) {
+ $c_callback = "${callback}_$name";
+ } else {
+ $f_more_sr->(sub {
+ my ($sr) = @_;
+ $cbs{$sr} .= " $c_rtype_callout (*${name})$c_decl;\n";
+ return
+ " const ".cbtype($sr)." *const cbs =\n".
+ " ${getcallbacks}_${sr}(user);\n";
+ });
+ $c_callback = "cbs->${name}";
+ }
+ my $c_make_callback = "$c_callback($c_callback_args)";
+ if ($flags !~ m/W/) {
+ $f_more_sr->(" $c_make_callback;\n");
+ } else {
+ $f_more_sr->(" r = $c_make_callback;\n".
+ " $sendreply(r, user);\n");
+ f_decl($sendreply, ''callout'', ''void'',
''(int r, void *user)'');
+ }
+ if ($flags =~ m/x/) {
+ my $c_v = "(1u<<$msgnum)";
+ my $c_cb = "cbs->$name";
+ $f_more_sr->(" if ($c_cb) cbflags |= $c_v;\n",
$enumcallbacks);
+ $f_more_sr->(" $c_cb = (cbflags & $c_v) ?
${encode}_${name} : 0;\n",
+ $setcallbacks);
+ }
+ $f_more_sr->(" return 1;\n }\n\n");
+ f_decl("${callback}_$name", ''callout'',
$c_rtype_callout, $c_decl);
+ f_decl("${encode}_$name", ''helper'',
$c_rtype_helper, $c_decl);
+ f_more("${encode}_$name",
+" if (buf) break;
+ buf = ${helper}_allocbuf(len, user);
+ assert(buf);
+ allocd = len;
+ len = 0;
+ }
+ assert(len == allocd);
+ ${transmit}(buf, len, user);
+");
+ if ($flags =~ m/W/) {
+ f_more("${encode}_$name", (<<END.($debug ? <<END :
'''').<<END));
+ int r = ${helper}_getreply(user);
+END
+ fprintf(stderr,"libxl-save-helper: $name got reply %d\\n",r);
+END
+ return r;
+END
+ }
+}
+
+print "/* AUTOGENERATED by $0 DO NOT EDIT */\n\n" or die $!;
+
+foreach my $sr (qw(save restore)) {
+ f_more("${enumcallbacks}_${sr}",
+ " return cbflags;\n");
+ f_more("${receiveds}_${sr}",
+ " default:\n".
+ " return 0;\n".
+ " }");
+ $cbs{$sr} .= "} ".cbtype($sr).";\n\n";
+ if ($ch eq ''h'') {
+ print $cbs{$sr} or die $!;
+ print "struct ${sr}_callbacks;\n";
+ }
+}
+
+if ($ch eq ''c'') {
+ foreach my $name (@outfuncs) {
+ next unless defined $func{$name};
+ $func{$name} .= "}\n\n";
+ $out_body{$func_ah{$name}} .= $func{$name};
+ delete $func{$name};
+ }
+ print $out_body{$want_ah} or die $!;
+} else {
+ foreach my $name (sort keys %out_decls) {
+ next unless $func_ah{$name} eq $want_ah;
+ print $out_decls{$name} or die $!;
+ }
+}
+
+close STDOUT or die $!;
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 06/19] libxl: rename libxl_dom:save_helper to physmap_path
"save_helper" isn''t very descriptive. Also it is now
confusing
because it reads like it might refer to the libxl-save-helper
executable which runs xc_domain_save and xc_domain_restore.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl_dom.c | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index fea0c94..a597627 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -734,7 +734,7 @@ int libxl__domain_suspend_common_callback(void *user)
return 1;
}
-static inline char *save_helper(libxl__gc *gc, uint32_t domid,
+static inline char *physmap_path(libxl__gc *gc, uint32_t domid,
char *phys_offset, char *node)
{
return libxl__sprintf(gc,
@@ -779,21 +779,21 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
return -1;
}
- xs_path = save_helper(gc, domid, phys_offset, "start_addr");
+ xs_path = physmap_path(gc, domid, phys_offset, "start_addr");
start_addr = libxl__xs_read(gc, 0, xs_path);
if (start_addr == NULL) {
LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
- xs_path = save_helper(gc, domid, phys_offset, "size");
+ xs_path = physmap_path(gc, domid, phys_offset, "size");
size = libxl__xs_read(gc, 0, xs_path);
if (size == NULL) {
LOG(ERROR, "%s is NULL", xs_path);
return -1;
}
- xs_path = save_helper(gc, domid, phys_offset, "name");
+ xs_path = physmap_path(gc, domid, phys_offset, "name");
name = libxl__xs_read(gc, 0, xs_path);
if (name == NULL)
namelen = 0;
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 07/19] libxl: provide libxl__xs_*_checked and libxl__xs_transaction_*
These useful utility functions make dealing with xenstore a little
less painful.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v3:
* Fixed typo `transacton'' in log messages.
---
tools/libxl/libxl_internal.h | 38 +++++++++++++++++++++
tools/libxl/libxl_xshelp.c | 76 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 114 insertions(+), 0 deletions(-)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index dbf7722..4634f5a 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -498,6 +498,44 @@ _hidden bool libxl__xs_mkdir(libxl__gc *gc,
xs_transaction_t t,
_hidden char *libxl__xs_libxl_path(libxl__gc *gc, uint32_t domid);
+
+/*----- "checked" xenstore access functions -----*/
+/* Each of these functions will check that it succeeded; if it
+ * fails it logs and returns ERROR_FAIL.
+ */
+
+/* On success, *result_out came from the gc.
+ * On error, *result_out is undefined.
+ * ENOENT counts as success but sets *result_out=0
+ */
+int libxl__xs_read_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char **result_out);
+
+/* Does not include a trailing null.
+ * May usefully be combined with GCSPRINTF if the format string
+ * behaviour of libxl__xs_write is desirable. */
+int libxl__xs_write_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char *string);
+
+/* ENOENT is not an error (even if the parent directories don''t exist)
*/
+int libxl__xs_rm_checked(libxl__gc *gc, xs_transaction_t t, const char *path);
+
+/* Transaction functions, best used together.
+ * The caller should initialise *t to 0 (XBT_NULL) before calling start.
+ * Each function leaves *t!=0 iff the transaction needs cleaning up.
+ *
+ * libxl__xs_transaction_commit returns:
+ * <0 failure - a libxl error code
+ * +1 commit conflict; transaction has been destroyed and caller
+ * must go round again (call _start again and retry)
+ * 0 commited successfully
+ */
+int libxl__xs_transaction_start(libxl__gc *gc, xs_transaction_t *t);
+int libxl__xs_transaction_commit(libxl__gc *gc, xs_transaction_t *t);
+void libxl__xs_transaction_abort(libxl__gc *gc, xs_transaction_t *t);
+
+
+
/*
* This is a recursive delete, from top to bottom. What this function does
* is remove empty folders that contained the deleted entry.
diff --git a/tools/libxl/libxl_xshelp.c b/tools/libxl/libxl_xshelp.c
index c5b5364..993f527 100644
--- a/tools/libxl/libxl_xshelp.c
+++ b/tools/libxl/libxl_xshelp.c
@@ -135,6 +135,82 @@ char *libxl__xs_libxl_path(libxl__gc *gc, uint32_t domid)
return s;
}
+int libxl__xs_read_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char **result_out)
+{
+ char *result = libxl__xs_read(gc, t, path);
+ if (!result) {
+ if (errno != ENOENT) {
+ LOGE(ERROR, "xenstore read failed: `%s''", path);
+ return ERROR_FAIL;
+ }
+ }
+ *result_out = result;
+ return 0;
+}
+
+int libxl__xs_write_checked(libxl__gc *gc, xs_transaction_t t,
+ const char *path, const char *string)
+{
+ size_t length = strlen(string);
+ if (!xs_write(CTX->xsh, t, path, string, length)) {
+ LOGE(ERROR, "xenstore write failed: `%s'' =
`%s''", path, string);
+ return ERROR_FAIL;
+ }
+ return 0;
+}
+
+int libxl__xs_rm_checked(libxl__gc *gc, xs_transaction_t t, const char *path)
+{
+ if (!xs_rm(CTX->xsh, t, path)) {
+ if (errno == ENOENT)
+ return 0;
+
+ LOGE(ERROR, "xenstore rm failed: `%s''", path);
+ return ERROR_FAIL;
+ }
+ return 0;
+}
+
+int libxl__xs_transaction_start(libxl__gc *gc, xs_transaction_t *t)
+{
+ assert(!*t);
+ *t = xs_transaction_start(CTX->xsh);
+ if (!*t) {
+ LOGE(ERROR, "could not create xenstore transaction");
+ return ERROR_FAIL;
+ }
+ return 0;
+}
+
+int libxl__xs_transaction_commit(libxl__gc *gc, xs_transaction_t *t)
+{
+ assert(*t);
+
+ if (!xs_transaction_end(CTX->xsh, *t, 0)) {
+ if (errno == EAGAIN)
+ return +1;
+
+ *t = 0;
+ LOGE(ERROR, "could not commit xenstore transaction");
+ return ERROR_FAIL;
+ }
+
+ *t = 0;
+ return 0;
+}
+
+void libxl__xs_transaction_abort(libxl__gc *gc, xs_transaction_t *t)
+{
+ if (!*t)
+ return;
+
+ if (!xs_transaction_end(CTX->xsh, *t, 1))
+ LOGE(ERROR, "could not abort xenstore transaction");
+
+ *t = 0;
+}
+
int libxl__xs_path_cleanup(libxl__gc *gc, xs_transaction_t t, char *user_path)
{
unsigned int nb = 0;
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 08/19] libxl: wait for qemu to acknowledge logdirty command
The current migration code in libxl instructs qemu to start or stop
logdirty, but it does not wait for an acknowledgement from qemu before
continuing. This might lead to memory corruption (!)
Fix this by waiting for qemu to acknowledge the command.
Unfortunately the necessary ao arrangements for waiting for this
command are unique because qemu has a special protocol for this
particular operation.
Also, this change means that the switch_qemu_logdirty callback
implementation in libxl can no longer synchronously produce its return
value, as it now needs to wait for xenstore. So we tell the
marshalling code generator that it is a message which does not need a
reply. This turns the callback function called by the marshaller into
one which returns void; the callback function arranges to later
explicitly sends the reply to the helper, when the xs watch triggers
and the appropriate value is read from xenstore.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tools/libxl/libxl_dom.c | 176 +++++++++++++++++++++++++++++++++---
tools/libxl/libxl_internal.h | 18 ++++-
tools/libxl/libxl_save_callout.c | 8 ++
tools/libxl/libxl_save_msgs_gen.pl | 7 +-
4 files changed, 193 insertions(+), 16 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index a597627..d5ac79f 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -528,30 +528,180 @@ int libxl__toolstack_restore(uint32_t domid, const
uint8_t *buf,
static void domain_suspend_done(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc);
-/*----- callbacks, called by xc_domain_save -----*/
+/*----- complicated callback, called by xc_domain_save -----*/
+
+/*
+ * We implement the other end of protocol for controlling qemu-dm''s
+ * logdirty. There is no documentation for this protocol, but our
+ * counterparty''s implementation is in
+ * qemu-xen-traditional.git:xenstore.c in the function
+ * xenstore_process_logdirty_event
+ */
+
+static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
+ const struct timeval *requested_abs);
+static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*,
+ const char *watch_path, const char *event_path);
+static void switch_logdirty_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int ok);
-int libxl__domain_suspend_common_switch_qemu_logdirty
+static void logdirty_init(libxl__logdirty_switch *lds)
+{
+ lds->cmd_path = 0;
+ libxl__ev_xswatch_init(&lds->watch);
+ libxl__ev_time_init(&lds->timeout);
+}
+
+void libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned enable, void *user)
{
libxl__save_helper_state *shs = user;
+ libxl__egc *egc = shs->egc;
libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__logdirty_switch *lds = &dss->logdirty;
STATE_AO_GC(dss->ao);
- char *path;
- bool rc;
+ int rc;
+ xs_transaction_t t = 0;
+ const char *got;
- path = libxl__sprintf(gc,
+ if (!lds->cmd_path) {
+ lds->cmd_path = GCSPRINTF(
"/local/domain/0/device-model/%u/logdirty/cmd",
domid);
- if (!path)
- return 1;
+ lds->ret_path = GCSPRINTF(
+ "/local/domain/0/device-model/%u/logdirty/ret",
domid);
+ }
+ lds->cmd = enable ? "enable" : "disable";
- if (enable)
- rc = xs_write(CTX->xsh, XBT_NULL, path, "enable",
strlen("enable"));
- else
- rc = xs_write(CTX->xsh, XBT_NULL, path, "disable",
strlen("disable"));
+ rc = libxl__ev_xswatch_register(gc, &lds->watch,
+ switch_logdirty_xswatch, lds->ret_path);
+ if (rc) goto out;
+
+ rc = libxl__ev_time_register_rel(gc, &lds->timeout,
+ switch_logdirty_timeout, 10*1000);
+ if (rc) goto out;
+
+ for (;;) {
+ rc = libxl__xs_transaction_start(gc, &t);
+ if (rc) goto out;
+
+ rc = libxl__xs_read_checked(gc, t, lds->cmd_path, &got);
+ if (rc) goto out;
+
+ if (got) {
+ const char *got_ret;
+ rc = libxl__xs_read_checked(gc, t, lds->ret_path, &got_ret);
+ if (rc) goto out;
- return rc ? 0 : 1;
+ if (strcmp(got, got_ret)) {
+ LOG(ERROR,"controlling logdirty: qemu was already
sent"
+ " command `%s'' (xenstore path `%s'')
but result is `%s''",
+ got, lds->cmd_path, got_ret ? got_ret :
"<none>");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ rc = libxl__xs_rm_checked(gc, t, lds->cmd_path);
+ if (rc) goto out;
+ }
+
+ rc = libxl__xs_rm_checked(gc, t, lds->ret_path);
+ if (rc) goto out;
+
+ rc = libxl__xs_write_checked(gc, t, lds->cmd_path, lds->cmd);
+ if (rc) goto out;
+
+ rc = libxl__xs_transaction_commit(gc, &t);
+ if (!rc) break;
+ if (rc<0) goto out;
+ }
+
+ /* OK, wait for some callback */
+ return;
+
+ out:
+ LOG(ERROR,"logdirty switch failed (rc=%d), aborting suspend",rc);
+ switch_logdirty_done(egc,dss,0);
+}
+
+static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
+ const struct timeval *requested_abs)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(ev, *dss,
logdirty.timeout);
+ STATE_AO_GC(dss->ao);
+ LOG(ERROR,"logdirty switch: wait for device model timed out");
+ switch_logdirty_done(egc,dss,0);
}
+static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch,
+ const char *watch_path, const char *event_path)
+{
+ libxl__domain_suspend_state *dss + CONTAINER_OF(watch, *dss,
logdirty.watch);
+ libxl__logdirty_switch *lds = &dss->logdirty;
+ STATE_AO_GC(dss->ao);
+ const char *got;
+ xs_transaction_t t = 0;
+ int rc;
+
+ for (;;) {
+ rc = libxl__xs_transaction_start(gc, &t);
+ if (rc) goto out;
+
+ rc = libxl__xs_read_checked(gc, t, lds->ret_path, &got);
+ if (rc) goto out;
+
+ if (!got) {
+ rc = +1;
+ goto out;
+ }
+
+ if (strcmp(got, lds->cmd)) {
+ LOG(ERROR,"logdirty switch: sent command `%s'' but got
reply `%s''"
+ " (xenstore paths `%s'' / `%s'')",
lds->cmd, got,
+ lds->cmd_path, lds->ret_path);
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ rc = libxl__xs_rm_checked(gc, t, lds->cmd_path);
+ if (rc) goto out;
+
+ rc = libxl__xs_rm_checked(gc, t, lds->ret_path);
+ if (rc) goto out;
+
+ rc = libxl__xs_transaction_commit(gc, &t);
+ if (!rc) break;
+ if (rc<0) goto out;
+ }
+
+ out:
+ /* rc < 0: error
+ * rc == 0: ok, we are done
+ * rc == +1: need to keep waiting
+ */
+ libxl__xs_transaction_abort(gc, &t);
+
+ if (!rc) {
+ switch_logdirty_done(egc,dss,1);
+ } else if (rc < 0) {
+ LOG(ERROR,"logdirty switch: response read failed
(rc=%d)",rc);
+ switch_logdirty_done(egc,dss,0);
+ }
+}
+
+static void switch_logdirty_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int ok)
+{
+ STATE_AO_GC(dss->ao);
+ libxl__logdirty_switch *lds = &dss->logdirty;
+
+ libxl__ev_xswatch_deregister(gc, &lds->watch);
+ libxl__ev_time_deregister(gc, &lds->timeout);
+
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs,
ok);
+}
+
+/*----- callbacks, called by xc_domain_save -----*/
+
int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid)
{
libxl_ctx *ctx = libxl__gc_owner(gc);
@@ -873,6 +1023,8 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
libxl__srm_save_autogen_callbacks *const callbacks
&dss->shs.callbacks.save.a;
+ logdirty_init(&dss->logdirty);
+
switch (type) {
case LIBXL_DOMAIN_TYPE_HVM: {
char *path;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 4634f5a..3b4d250 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1864,6 +1864,14 @@ typedef struct libxl__domain_suspend_state
libxl__domain_suspend_state;
typedef void libxl__domain_suspend_cb(libxl__egc*,
libxl__domain_suspend_state*, int rc);
+typedef struct libxl__logdirty_switch {
+ const char *cmd;
+ const char *cmd_path;
+ const char *ret_path;
+ libxl__ev_xswatch watch;
+ libxl__ev_time timeout;
+} libxl__logdirty_switch;
+
struct libxl__domain_suspend_state {
/* set by caller of libxl__domain_suspend */
libxl__ao *ao;
@@ -1883,6 +1891,7 @@ struct libxl__domain_suspend_state {
int guest_responded;
int interval; /* checkpoint interval (for Remus) */
libxl__save_helper_state shs;
+ libxl__logdirty_switch logdirty;
};
@@ -2013,8 +2022,15 @@ _hidden void libxl__xc_domain_save(libxl__egc*,
libxl__domain_suspend_state*,
_hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
int rc, int retval, int errnoval);
+/* Used by asynchronous callbacks: ie ones which xc regards as
+ * returning a value, but which we want to handle asynchronously.
+ * Such functions'' actual callback function return void in libxl
+ * When they are ready to indicate completion, they call this. */
+void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
+ libxl__save_helper_state *shs, int return_value);
+
_hidden int libxl__domain_suspend_common_callback(void *data);
-_hidden int libxl__domain_suspend_common_switch_qemu_logdirty
+_hidden void libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data);
_hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
uint32_t *len, void *data);
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index a39f434..ded311c 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -128,6 +128,14 @@ void libxl__xc_domain_save(libxl__egc *egc,
libxl__domain_suspend_state *dss,
}
+void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
+ libxl__save_helper_state *shs, int return_value)
+{
+ shs->egc = egc;
+ libxl__srm_callout_sendreply(return_value, shs);
+ shs->egc = 0;
+}
+
/*----- helper execution -----*/
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
diff --git a/tools/libxl/libxl_save_msgs_gen.pl
b/tools/libxl/libxl_save_msgs_gen.pl
index cd0837e..8832c46 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -14,6 +14,7 @@ our @msgs = (
# x - function pointer is in struct {save,restore}_callbacks
# and its null-ness needs to be passed through to the
helper''s xc
# W - needs a return value; callback is synchronous
+ # A - needs a return value; callback is asynchronous
[ 1, ''sr'', "log",
[qw(uint32_t level
uint32_t errnoval
STRING context
@@ -25,7 +26,7 @@ our @msgs = (
[ 3, ''scxW'', "suspend", [] ],
[ 4, ''scxW'', "postcopy", [] ],
[ 5, ''scxW'', "checkpoint", [] ],
- [ 6, ''scxW'', "switch_qemu_logdirty",
[qw(int domid
+ [ 6, ''scxA'', "switch_qemu_logdirty",
[qw(int domid
unsigned enable)] ],
# toolstack_save done entirely `by hand''
[ 7, ''rcxW'', "toolstack_restore",
[qw(uint32_t domid
@@ -260,7 +261,7 @@ foreach my $msginfo (@msgs) {
$f_more_sr->(" int r;\n");
}
- my $c_rtype_helper = $flags =~ m/W/ ? ''int'' :
''void'';
+ my $c_rtype_helper = $flags =~ m/[WA]/ ? ''int'' :
''void'';
my $c_rtype_callout = $flags =~ m/W/ ? ''int'' :
''void'';
my $c_decl = ''('';
my $c_callback_args = '''';
@@ -348,7 +349,7 @@ END
assert(len == allocd);
${transmit}(buf, len, user);
");
- if ($flags =~ m/W/) {
+ if ($flags =~ m/[WA]/) {
f_more("${encode}_$name", (<<END.($debug ? <<END :
'''').<<END));
int r = ${helper}_getreply(user);
END
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 09/19] libxl: datacopier: provide "prefix data" facility
This will be used to write the qemu data banner to the save/migration
stream.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v3:
* Clarified and added comments explaining the `immediately''
constraint and the lack of a reentrancy/threading hazard.
* Fixed subject line typo.
---
tools/libxl/libxl_aoutils.c | 22 ++++++++++++++++++++++
tools/libxl/libxl_internal.h | 6 ++++++
2 files changed, 28 insertions(+), 0 deletions(-)
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index ee0df57..7f8d6d3 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -74,6 +74,28 @@ static void datacopier_check_state(libxl__egc *egc,
libxl__datacopier_state *dc)
}
}
+void libxl__datacopier_prefixdata(libxl__egc *egc, libxl__datacopier_state *dc,
+ const void *data, size_t len)
+{
+ libxl__datacopier_buf *buf;
+ /*
+ * It is safe for this to be called immediately after _start, as
+ * is documented in the public comment. _start''s caller must have
+ * the ctx locked, so other threads don''t get to mess with the
+ * contents, and the fd events cannot happen reentrantly. So we
+ * are guaranteed to beat the first data from the read fd.
+ */
+
+ assert(len < dc->maxsz - dc->used);
+
+ buf = libxl__zalloc(0, sizeof(*buf) - sizeof(buf->buf) + len);
+ buf->used = len;
+ memcpy(buf->buf, data, len);
+
+ dc->used += len;
+ LIBXL_TAILQ_INSERT_TAIL(&dc->bufs, buf, entry);
+}
+
static void datacopier_readable(libxl__egc *egc, libxl__ev_fd *ev,
int fd, short events, short revents) {
libxl__datacopier_state *dc = CONTAINER_OF(ev, *dc, toread);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3b4d250..7e0ab11 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1811,6 +1811,12 @@ _hidden void
libxl__datacopier_init(libxl__datacopier_state *dc);
_hidden void libxl__datacopier_kill(libxl__datacopier_state *dc);
_hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
+/* Inserts literal data into the output stream. The data is copied.
+ * May safely be used only immediately after libxl__datacopier_start
+ * (before the ctx is unlocked). But may be called multiple times.
+ * NB exceeding maxsz will fail an assertion! */
+_hidden void libxl__datacopier_prefixdata(libxl__egc*,
libxl__datacopier_state*,
+ const void *data, size_t len);
/*----- Save/restore helper (used by creation and suspend) -----*/
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 10/19] libxl: prepare for asynchronous writing of qemu save file
* Combine the various calls to libxl__device_model_savefile into one
at the start of libxl__domain_suspend, storing the result in the
dss. Consequently a few functions take a dss instead of some or all
of their other arguments.
* Make libxl__domain_save_device_model''s API into an asynchronous
style which takes a callback. The function is, however, still
synchronous; it will be made actually async in the next patch.
* Consequently make libxl__remus_domain_checkpoint_callback into an
asynchronous callback.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tools/libxl/libxl_dom.c | 54 ++++++++++++++++++++++++++----------
tools/libxl/libxl_internal.h | 18 ++++++++++--
tools/libxl/libxl_save_msgs_gen.pl | 2 +-
3 files changed, 55 insertions(+), 19 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index d5ac79f..3a53f97 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -702,11 +702,13 @@ static void switch_logdirty_done(libxl__egc *egc,
/*----- callbacks, called by xc_domain_save -----*/
-int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid)
+int libxl__domain_suspend_device_model(libxl__gc *gc,
+ libxl__domain_suspend_state *dss)
{
libxl_ctx *ctx = libxl__gc_owner(gc);
int ret = 0;
- const char *filename = libxl__device_model_savefile(gc, domid);
+ uint32_t const domid = dss->domid;
+ const char *const filename = dss->dm_savefile;
switch (libxl__device_model_version_running(gc, domid)) {
case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL: {
@@ -875,7 +877,7 @@ int libxl__domain_suspend_common_callback(void *user)
guest_suspended:
if (dss->hvm) {
- ret = libxl__domain_suspend_device_model(gc, dss->domid);
+ ret = libxl__domain_suspend_device_model(gc, dss);
if (ret) {
LOG(ERROR, "libxl__domain_suspend_device_model failed
ret=%d", ret);
return 0;
@@ -990,19 +992,32 @@ static int libxl__remus_domain_resume_callback(void *data)
return 1;
}
-static int libxl__remus_domain_checkpoint_callback(void *data)
+/*----- remus asynchronous checkpoint callback -----*/
+
+static void remus_checkpoint_dm_saved(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int
rc);
+
+static void libxl__remus_domain_checkpoint_callback(void *data)
{
libxl__domain_suspend_state *dss = data;
+ libxl__egc *egc = dss->shs.egc;
STATE_AO_GC(dss->ao);
/* This would go into tailbuf. */
- if (dss->hvm &&
- libxl__domain_save_device_model(gc, dss->domid, dss->fd))
- return 0;
+ if (dss->hvm) {
+ libxl__domain_save_device_model(egc, dss, remus_checkpoint_dm_saved);
+ } else {
+ remus_checkpoint_dm_saved(egc, dss, 0);
+ }
+}
+static void remus_checkpoint_dm_saved(libxl__egc *egc,
+ libxl__domain_suspend_state *dss, int rc)
+{
/* TODO: Wait for disk and memory ack, release network buffer */
+ /* TODO: make this asynchronous */
usleep(dss->interval * 1000);
- return 1;
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
}
/*----- main code for suspending, in order of execution -----*/
@@ -1052,6 +1067,7 @@ void libxl__domain_suspend(libxl__egc *egc,
libxl__domain_suspend_state *dss)
dss->suspend_eventchn = -1;
dss->guest_responded = 0;
+ dss->dm_savefile = libxl__device_model_savefile(gc, domid);
if (r_info != NULL) {
dss->interval = r_info->interval;
@@ -1101,7 +1117,6 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void
*dss_void,
/* Convenience aliases */
const libxl_domain_type type = dss->type;
- const uint32_t domid = dss->domid;
if (rc)
goto out;
@@ -1119,11 +1134,11 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void
*dss_void,
}
if (type == LIBXL_DOMAIN_TYPE_HVM) {
- rc = libxl__domain_suspend_device_model(gc, domid);
+ rc = libxl__domain_suspend_device_model(gc, dss);
if (rc) goto out;
- rc = libxl__domain_save_device_model(gc, domid, dss->fd);
- if (rc) goto out;
+ libxl__domain_save_device_model(egc, dss, domain_suspend_done);
+ return;
}
rc = 0;
@@ -1132,14 +1147,22 @@ out:
domain_suspend_done(egc, dss, rc);
}
-int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd)
+void libxl__domain_save_device_model(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ libxl__save_device_model_cb *callback)
{
+ STATE_AO_GC(dss->ao);
int rc, fd2 = -1, c;
char buf[1024];
- const char *filename = libxl__device_model_savefile(gc, domid);
struct stat st;
uint32_t qemu_state_len;
+ dss->save_dm_callback = callback;
+
+ /* Convenience aliases */
+ const char *const filename = dss->dm_savefile;
+ const int fd = dss->fd;
+
if (stat(filename, &st) < 0)
{
LOG(ERROR, "Unable to stat qemu save file\n");
@@ -1181,7 +1204,8 @@ int libxl__domain_save_device_model(libxl__gc *gc,
uint32_t domid, int fd)
out:
if (fd2 >= 0) close(fd2);
unlink(filename);
- return rc;
+
+ dss->save_dm_callback(egc, dss, rc);
}
static void domain_suspend_done(libxl__egc *egc,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 7e0ab11..c7fe9e9 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -824,10 +824,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t
domid,
_hidden int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
uint32_t size, void *data);
-_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
-_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
-_hidden int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int
fd);
+
_hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid);
_hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid);
@@ -1869,6 +1867,8 @@ typedef struct libxl__domain_suspend_state
libxl__domain_suspend_state;
typedef void libxl__domain_suspend_cb(libxl__egc*,
libxl__domain_suspend_state*, int rc);
+typedef void libxl__save_device_model_cb(libxl__egc*,
+ libxl__domain_suspend_state*, int rc);
typedef struct libxl__logdirty_switch {
const char *cmd;
@@ -1895,9 +1895,12 @@ struct libxl__domain_suspend_state {
int hvm;
int xcflags;
int guest_responded;
+ const char *dm_savefile;
int interval; /* checkpoint interval (for Remus) */
libxl__save_helper_state shs;
libxl__logdirty_switch logdirty;
+ /* private for libxl__domain_save_device_model */
+ libxl__save_device_model_cb *save_dm_callback;
};
@@ -2053,6 +2056,15 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc,
_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int rc, int retval, int errnoval);
+/* Each time the dm needs to be saved, we must call suspend and then save */
+_hidden int libxl__domain_suspend_device_model(libxl__gc *gc,
+ libxl__domain_suspend_state *dss);
+_hidden void libxl__domain_save_device_model(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ libxl__save_device_model_cb *callback);
+
+_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t
domid);
+
/*
* Convenience macros.
diff --git a/tools/libxl/libxl_save_msgs_gen.pl
b/tools/libxl/libxl_save_msgs_gen.pl
index 8832c46..3ac6f80 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -25,7 +25,7 @@ our @msgs = (
''unsigned
long'', ''total''] ],
[ 3, ''scxW'', "suspend", [] ],
[ 4, ''scxW'', "postcopy", [] ],
- [ 5, ''scxW'', "checkpoint", [] ],
+ [ 5, ''scxA'', "checkpoint", [] ],
[ 6, ''scxA'', "switch_qemu_logdirty",
[qw(int domid
unsigned enable)] ],
# toolstack_save done entirely `by hand''
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 11/19] libxl: Make libxl__domain_save_device_model asynchronous
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in series v3:
* Improve one more a debugging message.
---
tools/libxl/libxl_dom.c | 100 +++++++++++++++++++++++++++---------------
tools/libxl/libxl_internal.h | 1 +
2 files changed, 66 insertions(+), 35 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 3a53f97..14eb77e 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1147,15 +1147,17 @@ out:
domain_suspend_done(egc, dss, rc);
}
+static void save_device_model_datacopier_done(libxl__egc *egc,
+ libxl__datacopier_state *dc, int onwrite, int errnoval);
+
void libxl__domain_save_device_model(libxl__egc *egc,
libxl__domain_suspend_state *dss,
libxl__save_device_model_cb *callback)
{
STATE_AO_GC(dss->ao);
- int rc, fd2 = -1, c;
- char buf[1024];
struct stat st;
uint32_t qemu_state_len;
+ int rc;
dss->save_dm_callback = callback;
@@ -1163,49 +1165,77 @@ void libxl__domain_save_device_model(libxl__egc *egc,
const char *const filename = dss->dm_savefile;
const int fd = dss->fd;
- if (stat(filename, &st) < 0)
+ libxl__datacopier_state *dc = &dss->save_dm_datacopier;
+ memset(dc, 0, sizeof(*dc));
+ dc->readwhat = GCSPRINTF("qemu save file %s", filename);
+ dc->ao = ao;
+ dc->readfd = -1;
+ dc->writefd = fd;
+ dc->maxsz = INT_MAX;
+ dc->copywhat = GCSPRINTF("qemu save file for domain %"PRIu32,
dss->domid);
+ dc->writewhat = "save/migration stream";
+ dc->callback = save_device_model_datacopier_done;
+
+ dc->readfd = open(filename, O_RDONLY);
+ if (dc->readfd < 0) {
+ LOGE(ERROR, "unable to open %s", dc->readwhat);
+ goto out;
+ }
+
+ if (fstat(dc->readfd, &st))
{
- LOG(ERROR, "Unable to stat qemu save file\n");
- rc = ERROR_FAIL;
+ LOGE(ERROR, "unable to fstat %s", dc->readwhat);
+ goto out;
+ }
+
+ if (!S_ISREG(st.st_mode)) {
+ LOG(ERROR, "%s is not a plain file!", dc->readwhat);
goto out;
}
qemu_state_len = st.st_size;
- LOG(DEBUG, "Qemu state is %d bytes\n", qemu_state_len);
+ LOG(DEBUG, "%s is %d bytes", dc->readwhat, qemu_state_len);
- rc = libxl_write_exactly(CTX, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE),
- "saved-state file", "qemu
signature");
- if (rc)
- goto out;
+ rc = libxl__datacopier_start(dc);
+ if (rc) goto out;
- rc = libxl_write_exactly(CTX, fd, &qemu_state_len,
sizeof(qemu_state_len),
- "saved-state file", "saved-state
length");
- if (rc)
- goto out;
+ libxl__datacopier_prefixdata(egc, dc,
+ QEMU_SIGNATURE, strlen(QEMU_SIGNATURE));
- fd2 = open(filename, O_RDONLY);
- if (fd2 < 0) {
- LOGE(ERROR, "Unable to open qemu save file\n");
- goto out;
- }
- while ((c = read(fd2, buf, sizeof(buf))) != 0) {
- if (c < 0) {
- if (errno == EINTR)
- continue;
- rc = errno;
- goto out;
- }
- rc = libxl_write_exactly(
- CTX, fd, buf, c, "saved-state file", "qemu
state");
- if (rc)
- goto out;
+ libxl__datacopier_prefixdata(egc, dc,
+ &qemu_state_len, sizeof(qemu_state_len));
+ return;
+
+ out:
+ save_device_model_datacopier_done(egc, dc, -1, 0);
+}
+
+static void save_device_model_datacopier_done(libxl__egc *egc,
+ libxl__datacopier_state *dc, int onwrite, int errnoval)
+{
+ libxl__domain_suspend_state *dss + CONTAINER_OF(dc, *dss,
save_dm_datacopier);
+ STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases */
+ const char *const filename = dss->dm_savefile;
+ int our_rc = 0;
+ int rc;
+
+ libxl__datacopier_kill(dc);
+
+ if (onwrite || errnoval)
+ our_rc = ERROR_FAIL;
+
+ if (dc->readfd >= 0) {
+ close(dc->readfd);
+ dc->readfd = -1;
}
- rc = 0;
-out:
- if (fd2 >= 0) close(fd2);
- unlink(filename);
- dss->save_dm_callback(egc, dss, rc);
+ rc = libxl__remove_file(gc, filename);
+ if (!our_rc) our_rc = rc;
+
+ dss->save_dm_callback(egc, dss, our_rc);
}
static void domain_suspend_done(libxl__egc *egc,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index c7fe9e9..f90814a 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1901,6 +1901,7 @@ struct libxl__domain_suspend_state {
libxl__logdirty_switch logdirty;
/* private for libxl__domain_save_device_model */
libxl__save_device_model_cb *save_dm_callback;
+ libxl__datacopier_state save_dm_datacopier;
};
--
1.7.2.5
In the next patch we are going to change the definition of NOGC to
require a local variable libxl__gc *gc.
libxl_get_cpu_topology doesn''t have one but does use NOGC.
Fix this by:
- introducing an `out'' label
- replacing the only call to `return'' with a suitable assignment
to ret and a `goto out''.
- adding uses of GC_INIT and GC_FREE.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tools/libxl/libxl.c | 6 +++++-
1 files changed, 5 insertions(+), 1 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 7de026b..2a31528 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -3202,6 +3202,7 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo
*physinfo)
libxl_cputopology *libxl_get_cpu_topology(libxl_ctx *ctx, int *nr)
{
+ GC_INIT(ctx);
xc_topologyinfo_t tinfo;
DECLARE_HYPERCALL_BUFFER(xc_cpu_to_core_t, coremap);
DECLARE_HYPERCALL_BUFFER(xc_cpu_to_socket_t, socketmap);
@@ -3214,7 +3215,8 @@ libxl_cputopology *libxl_get_cpu_topology(libxl_ctx *ctx,
int *nr)
if (max_cpus == 0)
{
LIBXL__LOG(ctx, XTL_ERROR, "Unable to determine number of
CPUS");
- return NULL;
+ ret = NULL;
+ goto out;
}
coremap = xc_hypercall_buffer_alloc
@@ -3259,6 +3261,8 @@ fail:
if (ret)
*nr = max_cpus;
+ out:
+ GC_FREE;
return ret;
}
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 13/19] libxl: Do not pass NULL as gc_opt; introduce NOGC
In 25182:6c3345d7e9d9 the practice of passing NULL to gc-using memory
allocation functions was introduced. However, the arrangements there
were not correct as committed, because the error handling and logging
depends on getting a ctx from the gc - so an allocation error would in
fact result in libxl dereferencing NULL.
Instead, provide a special dummy gc in the ctx, called `nogc_gc''. It
is marked out specially by having alloc_maxsize==-1, which is
otherwise invalid.
Functions which need to actually look into the gc use the new test
function gc_is_real (whose purpose is mainly clarity of the code) to
check whether the gc is the dummy one, and do nothing if it is. And
we provide a helper macro NOGC which uses the in-scope real gc to find
the ctx and hence the dummy gc (and which replaces the previous
#define NOGC NULL).
Change all callers which pass 0 or NULL to an allocation function to
use NOGC or &ctx->nogc_gc, as applicable in the context.
We add a comment near the definition of LIBXL_INIT_GC pointing out
that it isn''t any more the only place a libxl__gc struct is
initialised, for the benefit of anyone changing the contents of gc''s
in the future.
Also, actually document that libxl__ptr_add is legal with ptr==NULL,
and change a couple of calls not to check for NULL argument.
Reported-by: Bamvor Jian Zhang <bjzhang@suse.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Bamvor Jian Zhang <bjzhang@suse.com>
Cc: Ian Campbell <Ian.Campbell@citrix.com>
---
tools/libxl/libxl.c | 3 +++
tools/libxl/libxl_aoutils.c | 3 ++-
tools/libxl/libxl_create.c | 2 +-
tools/libxl/libxl_event.c | 5 +++--
tools/libxl/libxl_exec.c | 2 +-
tools/libxl/libxl_fork.c | 2 +-
tools/libxl/libxl_internal.c | 11 +++++++++--
tools/libxl/libxl_internal.h | 37 +++++++++++++++++++++++--------------
tools/libxl/libxl_utils.c | 6 ++----
tools/libxl/libxl_xshelp.c | 7 ++-----
10 files changed, 47 insertions(+), 31 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 2a31528..a257902 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -41,6 +41,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
/* First initialise pointers etc. (cannot fail) */
+ ctx->nogc_gc.alloc_maxsize = -1;
+ ctx->nogc_gc.owner = ctx;
+
LIBXL_TAILQ_INIT(&ctx->occurred);
ctx->osevent_hooks = 0;
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index 7f8d6d3..99972a2 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -77,6 +77,7 @@ static void datacopier_check_state(libxl__egc *egc,
libxl__datacopier_state *dc)
void libxl__datacopier_prefixdata(libxl__egc *egc, libxl__datacopier_state *dc,
const void *data, size_t len)
{
+ EGC_GC;
libxl__datacopier_buf *buf;
/*
* It is safe for this to be called immediately after _start, as
@@ -88,7 +89,7 @@ void libxl__datacopier_prefixdata(libxl__egc *egc,
libxl__datacopier_state *dc,
assert(len < dc->maxsz - dc->used);
- buf = libxl__zalloc(0, sizeof(*buf) - sizeof(buf->buf) + len);
+ buf = libxl__zalloc(NOGC, sizeof(*buf) - sizeof(buf->buf) + len);
buf->used = len;
memcpy(buf->buf, data, len);
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 00705d8..ab000bc 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -108,7 +108,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
}
if (b_info->blkdev_start == NULL)
- b_info->blkdev_start = libxl__strdup(0, "xvda");
+ b_info->blkdev_start = libxl__strdup(NOGC, "xvda");
if (b_info->type == LIBXL_DOMAIN_TYPE_HVM) {
if (!b_info->u.hvm.bios)
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 565d2c2..eb23a93 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -772,7 +772,7 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller
*poller,
if (poller->fd_rindices_allocd < maxfd) {
assert(ARRAY_SIZE_OK(poller->fd_rindices, maxfd));
poller->fd_rindices - libxl__realloc(0,
poller->fd_rindices,
+ libxl__realloc(NOGC, poller->fd_rindices,
maxfd * sizeof(*poller->fd_rindices));
memset(poller->fd_rindices + poller->fd_rindices_allocd,
0,
@@ -1099,9 +1099,10 @@ void libxl_event_free(libxl_ctx *ctx, libxl_event *event)
libxl_event *libxl__event_new(libxl__egc *egc,
libxl_event_type type, uint32_t domid)
{
+ EGC_GC;
libxl_event *ev;
- ev = libxl__zalloc(0,sizeof(*ev));
+ ev = libxl__zalloc(NOGC,sizeof(*ev));
ev->type = type;
ev->domid = domid;
diff --git a/tools/libxl/libxl_exec.c b/tools/libxl/libxl_exec.c
index 082bf44..cfa379c 100644
--- a/tools/libxl/libxl_exec.c
+++ b/tools/libxl/libxl_exec.c
@@ -280,7 +280,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
int status, rc;
libxl__spawn_init(ss);
- ss->ssd = libxl__zalloc(0, sizeof(*ss->ssd));
+ ss->ssd = libxl__zalloc(NOGC, sizeof(*ss->ssd));
libxl__ev_child_init(&ss->ssd->mid);
rc = libxl__ev_time_register_rel(gc, &ss->timeout,
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
index 9ff99e0..044ddad 100644
--- a/tools/libxl/libxl_fork.c
+++ b/tools/libxl/libxl_fork.c
@@ -92,7 +92,7 @@ libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
libxl__carefd *cf = 0;
libxl_fd_set_cloexec(ctx, fd, 1);
- cf = libxl__zalloc(NULL, sizeof(*cf));
+ cf = libxl__zalloc(&ctx->nogc_gc, sizeof(*cf));
cf->fd = fd;
LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry);
return cf;
diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
index 8139520..fbff7d0 100644
--- a/tools/libxl/libxl_internal.c
+++ b/tools/libxl/libxl_internal.c
@@ -30,11 +30,16 @@ void libxl__alloc_failed(libxl_ctx *ctx, const char *func,
#undef L
}
+static int gc_is_real(const libxl__gc *gc)
+{
+ return gc->alloc_maxsize >= 0;
+}
+
void libxl__ptr_add(libxl__gc *gc, void *ptr)
{
int i;
- if (!gc)
+ if (!gc_is_real(gc))
return;
if (!ptr)
@@ -66,6 +71,8 @@ void libxl__free_all(libxl__gc *gc)
void *ptr;
int i;
+ assert(gc_is_real(gc));
+
for (i = 0; i < gc->alloc_maxsize; i++) {
ptr = gc->alloc_ptrs[i];
gc->alloc_ptrs[i] = NULL;
@@ -104,7 +111,7 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t
new_size)
if (ptr == NULL) {
libxl__ptr_add(gc, new_ptr);
- } else if (new_ptr != ptr && gc != NULL) {
+ } else if (new_ptr != ptr && gc_is_real(gc)) {
for (i = 0; i < gc->alloc_maxsize; i++) {
if (gc->alloc_ptrs[i] == ptr) {
gc->alloc_ptrs[i] = new_ptr;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index f90814a..e69482c 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -277,10 +277,18 @@ struct libxl__poller {
int wakeup_pipe[2]; /* 0 means no fd allocated */
};
+struct libxl__gc {
+ /* mini-GC */
+ int alloc_maxsize; /* -1 means this is the dummy non-gc gc */
+ void **alloc_ptrs;
+ libxl_ctx *owner;
+};
+
struct libxl__ctx {
xentoollog_logger *lg;
xc_interface *xch;
struct xs_handle *xsh;
+ libxl__gc nogc_gc;
const libxl_event_hooks *event_hooks;
void *event_hooks_user;
@@ -356,13 +364,6 @@ typedef struct {
#define PRINTF_ATTRIBUTE(x, y) __attribute__((format(printf, x, y)))
-struct libxl__gc {
- /* mini-GC */
- int alloc_maxsize;
- void **alloc_ptrs;
- libxl_ctx *owner;
-};
-
struct libxl__egc {
/* For event-generating functions only.
* The egc and its gc may be accessed only on the creating thread. */
@@ -420,6 +421,7 @@ struct libxl__ao {
(gc).alloc_ptrs = 0; \
(gc).owner = (ctx); \
} while(0)
+ /* NB, also, a gc struct ctx->nogc_gc is initialised in libxl_ctx_alloc
*/
static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
{
@@ -438,13 +440,20 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
* All pointers returned by these functions are registered for garbage
* collection on exit from the outermost libxl callframe.
*
- * However, where the argument is stated to be "gc_opt", NULL may be
- * passed instead, in which case no garbage collection will occur; the
- * pointer must later be freed with free(). This is for memory
- * allocations of types (b) and (c).
+ * However, where the argument is stated to be "gc_opt",
&ctx->nogc_gc
+ * may be passed instead, in which case no garbage collection will
+ * occur; the pointer must later be freed with free(). (Passing NULL
+ * for gc_opt is not permitted.) This is for memory allocations of
+ * types (b) and (c). The convenience macro NOGC should be used where
+ * possible.
+ *
+ * NOGC (and ctx->nogc_gc) may ONLY be used with functions which
+ * explicitly declare that it''s OK. Use with nonconsenting functions
+ * may result in leaks of those functions'' internal allocations on the
+ * psuedo-gc.
*/
-/* register @ptr in @gc for free on exit from outermost libxl callframe. */
-_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr);
+/* register ptr in gc for free on exit from outermost libxl callframe. */
+_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */);
/* if this is the outermost libxl callframe then free all pointers in @gc */
_hidden void libxl__free_all(libxl__gc *gc);
/* allocate and zero @bytes. (similar to a gc''d malloc(3)+memzero())
*/
@@ -2110,7 +2119,7 @@ _hidden const char *libxl__device_model_savefile(libxl__gc
*gc, uint32_t domid);
#define GC_INIT(ctx) libxl__gc gc[1]; LIBXL_INIT_GC(gc[0],ctx)
#define GC_FREE libxl__free_all(gc)
#define CTX libxl__gc_owner(gc)
-#define NOGC NULL
+#define NOGC (&CTX->nogc_gc) /* pass only to consenting
functions */
/* Allocation macros all of which use the gc. */
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index 67ef82c..08c7dac 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -58,8 +58,7 @@ char *libxl_domid_to_name(libxl_ctx *ctx, uint32_t domid)
char *libxl__domid_to_name(libxl__gc *gc, uint32_t domid)
{
char *s = libxl_domid_to_name(libxl__gc_owner(gc), domid);
- if ( s )
- libxl__ptr_add(gc, s);
+ libxl__ptr_add(gc, s);
return s;
}
@@ -107,8 +106,7 @@ char *libxl_cpupoolid_to_name(libxl_ctx *ctx, uint32_t
poolid)
char *libxl__cpupoolid_to_name(libxl__gc *gc, uint32_t poolid)
{
char *s = libxl_cpupoolid_to_name(libxl__gc_owner(gc), poolid);
- if ( s )
- libxl__ptr_add(gc, s);
+ libxl__ptr_add(gc, s);
return s;
}
diff --git a/tools/libxl/libxl_xshelp.c b/tools/libxl/libxl_xshelp.c
index 993f527..7fdf164 100644
--- a/tools/libxl/libxl_xshelp.c
+++ b/tools/libxl/libxl_xshelp.c
@@ -86,11 +86,8 @@ char * libxl__xs_read(libxl__gc *gc, xs_transaction_t t,
const char *path)
char *ptr;
ptr = xs_read(ctx->xsh, t, path, NULL);
- if (ptr != NULL) {
- libxl__ptr_add(gc, ptr);
- return ptr;
- }
- return 0;
+ libxl__ptr_add(gc, ptr);
+ return ptr;
}
char *libxl__xs_get_dompath(libxl__gc *gc, uint32_t domid)
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 14/19] libxl: Get compiler to warn about gc_opt==NULL
Since it used to be legal to pass gc_opt==NULL, and there are various patches floating about and under developmetn which do so, add a compiler annotation which makes the build fail when that is done. This turns a runtime crash into a build failure, and should ensure that we don''t accidentally commit a broken combination of patches. This is something of an annoying approach because it adds a macro invocation to the RHS of every declaration of a function taking a gc_opt. So it should be reverted after Xen 4.2rc1. Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> --- tools/libxl/libxl_internal.h | 21 +++++++++++++-------- 1 files changed, 13 insertions(+), 8 deletions(-) diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index e69482c..8635764 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -453,28 +453,33 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc) * psuedo-gc. */ /* register ptr in gc for free on exit from outermost libxl callframe. */ -_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */); + +#define NN1 __attribute__((nonnull(1))) + /* It used to be legal to pass NULL for gc_opt. Get the compiler to + * warn about this if any slip through. */ + +_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */) NN1; /* if this is the outermost libxl callframe then free all pointers in @gc */ _hidden void libxl__free_all(libxl__gc *gc); /* allocate and zero @bytes. (similar to a gc''d malloc(3)+memzero()) */ -_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes); +_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes) NN1; /* allocate and zero memory for an array of @nmemb members of @size each. * (similar to a gc''d calloc(3)). */ -_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size); +_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size) NN1; /* change the size of the memory block pointed to by @ptr to @new_size bytes. * unlike other allocation functions here any additional space between the * oldsize and @new_size is not initialised (similar to a gc''d realloc(3)). */ -_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size); +_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size) NN1; /* print @fmt into an allocated string large enoughto contain the result. * (similar to gc''d asprintf(3)). */ -_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3); +_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3) NN1; /* duplicate the string @c (similar to a gc''d strdup(3)). */ -_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c); +_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c) NN1; /* duplicate at most @n bytes of string @c (similar to a gc''d strndup(3)). */ -_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n); +_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n) NN1; /* strip the last path component from @s and return as a newly allocated * string. (similar to a gc''d dirname(3)). */ -_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s); +_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s) NN1; /* Each of these logs errors and returns a libxl error code. * They do not mind if path is already removed. -- 1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 15/19] xl: Handle return value from libxl_domain_suspend correctly
libxl_domain_suspend returns a libxl error code. So it must be
wrapped with MUST and not CHK_ERRNO.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/xl_cmdimpl.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 19daa1c..a9125cc 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -2846,7 +2846,7 @@ static int save_domain(const char *p, const char
*filename, int checkpoint,
save_domain_core_writeconfig(fd, filename, config_data, config_len);
- CHK_ERRNO(libxl_domain_suspend(ctx, domid, fd, 0, NULL));
+ MUST(libxl_domain_suspend(ctx, domid, fd, 0, NULL));
close(fd);
if (checkpoint)
--
1.7.2.5
This was allocated using asprintf but never freed. Use GCSPRINTF.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl_create.c | 3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index ab000bc..5618293 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -740,9 +740,8 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void
*dcs_void,
goto out;
if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
- ret = asprintf(&state->saved_state,
+ state->saved_state = GCSPRINTF(
XC_DEVICE_MODEL_RESTORE_FILE".%d", domid);
- ret = (ret < 0) ? ERROR_FAIL : 0;
}
out:
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 17/19] libxl: do not leak spawned middle children
libxl__spawn_spawn would, when libxl__spawn_detach was called, make
the spawn become idle immediately. However it still has a child
process which needs to be waited for: the `detachable'' spawned
child.
This is wrong because the ultimate in-libxl caller may return to the
application, with a child process still forked but not reaped libxl
contrary to the documented behaviour of libxl.
Instead, replace libxl__spawn_detach with libxl__spawn_initiate_detach
which is asynchronous. The detachable spawned children are abolished;
instead, we defer calling back to the in-libxl user until the middle
child has been reaped.
Also, remove erroneous comment suggesting that `death'' callback
parameter to libxl__ev_child_fork may be NULL. It may not, and there
are no callers which pass NULL.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Changes in v3 of series:
* Now also remove erroneous comment about NULL fork death callback.
---
tools/libxl/libxl_dm.c | 14 ++++-
tools/libxl/libxl_exec.c | 124 ++++++++++++++++++++++++------------------
tools/libxl/libxl_internal.h | 43 ++++++++-------
3 files changed, 104 insertions(+), 77 deletions(-)
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 340fcfa..b3de145 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -908,6 +908,8 @@ static void device_model_confirm(libxl__egc *egc,
libxl__spawn_state *spawn,
const char *xsdata);
static void device_model_startup_failed(libxl__egc *egc,
libxl__spawn_state *spawn);
+static void device_model_detached(libxl__egc *egc,
+ libxl__spawn_state *spawn);
/* our "next step" function, called from those callbacks and
elsewhere */
static void device_model_spawn_outcome(libxl__egc *egc,
@@ -1015,6 +1017,7 @@ retry_transaction:
spawn->midproc_cb = libxl__spawn_record_pid;
spawn->confirm_cb = device_model_confirm;
spawn->failure_cb = device_model_startup_failed;
+ spawn->detached_cb = device_model_detached;
rc = libxl__spawn_spawn(egc, spawn);
if (rc < 0)
@@ -1048,9 +1051,7 @@ static void device_model_confirm(libxl__egc *egc,
libxl__spawn_state *spawn,
if (strcmp(xsdata, "running"))
return;
- libxl__spawn_detach(gc, spawn);
-
- device_model_spawn_outcome(egc, dmss, 0);
+ libxl__spawn_initiate_detach(gc, spawn);
}
static void device_model_startup_failed(libxl__egc *egc,
@@ -1060,6 +1061,13 @@ static void device_model_startup_failed(libxl__egc *egc,
device_model_spawn_outcome(egc, dmss, ERROR_FAIL);
}
+static void device_model_detached(libxl__egc *egc,
+ libxl__spawn_state *spawn)
+{
+ libxl__dm_spawn_state *dmss = CONTAINER_OF(spawn, *dmss, spawn);
+ device_model_spawn_outcome(egc, dmss, 0);
+}
+
static void device_model_spawn_outcome(libxl__egc *egc,
libxl__dm_spawn_state *dmss,
int rc)
diff --git a/tools/libxl/libxl_exec.c b/tools/libxl/libxl_exec.c
index cfa379c..bb85682 100644
--- a/tools/libxl/libxl_exec.c
+++ b/tools/libxl/libxl_exec.c
@@ -238,15 +238,16 @@ err:
/*
* Full set of possible states of a libxl__spawn_state and its _detachable:
*
- * ss-> ss-> ss-> | ssd->
ssd->
- * timeout xswatch ssd | mid ss
- * - Undefined undef undef no | - -
- * - Idle Idle Idle no | - -
- * - Active Active Active yes | Active yes
- * - Partial Active/Idle Active/Idle maybe | Active/Idle yes (if
exists)
- * - Detached - - - | Active no
+ * detaching failed mid timeout xswatch
+ * - Undefined undef undef - undef undef
+ * - Idle any any Idle Idle Idle
+ * - Attached OK 0 0 Active Active Active
+ * - Attached Failed 0 1 Active Idle Idle
+ * - Detaching 1 maybe Active Idle Idle
+ * - Partial any any Idle Active/Idle Active/Idle
*
- * When in state Detached, the middle process has been sent a SIGKILL.
+ * When in states Detaching or Attached Failed, the middle process has
+ * been sent a SIGKILL.
*/
/* Event callbacks. */
@@ -257,19 +258,18 @@ static void spawn_timeout(libxl__egc *egc, libxl__ev_time
*ev,
static void spawn_middle_death(libxl__egc *egc, libxl__ev_child *childw,
pid_t pid, int status);
-/* Precondition: Partial. Results: Detached. */
+/* Precondition: Partial. Results: Idle. */
static void spawn_cleanup(libxl__gc *gc, libxl__spawn_state *ss);
-/* Precondition: Partial; caller has logged failure reason.
- * Results: Caller notified of failure;
- * after return, ss may be completely invalid as caller may reuse it */
-static void spawn_failed(libxl__egc *egc, libxl__spawn_state *ss);
+/* Precondition: Attached or Detaching; caller has logged failure reason.
+ * Results: Detaching, or Attached Failing */
+static void spawn_fail(libxl__egc *egc, libxl__spawn_state *ss);
void libxl__spawn_init(libxl__spawn_state *ss)
{
+ libxl__ev_child_init(&ss->mid);
libxl__ev_time_init(&ss->timeout);
libxl__ev_xswatch_init(&ss->xswatch);
- ss->ssd = 0;
}
int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *ss)
@@ -280,8 +280,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
int status, rc;
libxl__spawn_init(ss);
- ss->ssd = libxl__zalloc(NOGC, sizeof(*ss->ssd));
- libxl__ev_child_init(&ss->ssd->mid);
+ ss->failed = ss->detaching = 0;
rc = libxl__ev_time_register_rel(gc, &ss->timeout,
spawn_timeout, ss->timeout_ms);
@@ -291,7 +290,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
spawn_watch_event, ss->xspath);
if (rc) goto out_err;
- pid_t middle = libxl__ev_child_fork(gc, &ss->ssd->mid,
spawn_middle_death);
+ pid_t middle = libxl__ev_child_fork(gc, &ss->mid,
spawn_middle_death);
if (middle ==-1) { rc = ERROR_FAIL; goto out_err; }
if (middle) {
@@ -344,54 +343,64 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state
*ss)
static void spawn_cleanup(libxl__gc *gc, libxl__spawn_state *ss)
{
+ assert(!libxl__ev_child_inuse(&ss->mid));
+ libxl__ev_time_deregister(gc, &ss->timeout);
+ libxl__ev_xswatch_deregister(gc, &ss->xswatch);
+}
+
+static void spawn_detach(libxl__gc *gc, libxl__spawn_state *ss)
+/* Precondition: Attached or Detaching, but caller must have just set
+ * at least one of detaching or failed.
+ * Results: Detaching or Attached Failed */
+{
int r;
+ assert(libxl__ev_child_inuse(&ss->mid));
libxl__ev_time_deregister(gc, &ss->timeout);
libxl__ev_xswatch_deregister(gc, &ss->xswatch);
- libxl__spawn_state_detachable *ssd = ss->ssd;
- if (ssd) {
- if (libxl__ev_child_inuse(&ssd->mid)) {
- pid_t child = ssd->mid.pid;
- r = kill(child, SIGKILL);
- if (r && errno != ESRCH)
- LOGE(WARN, "%s: failed to kill intermediate child
(pid=%lu)",
- ss->what, (unsigned long)child);
- }
+ pid_t child = ss->mid.pid;
+ r = kill(child, SIGKILL);
+ if (r && errno != ESRCH)
+ LOGE(WARN, "%s: failed to kill intermediate child (pid=%lu)",
+ ss->what, (unsigned long)child);
+}
- /* disconnect the ss and ssd from each other */
- ssd->ss = 0;
- ss->ssd = 0;
- }
+void libxl__spawn_initiate_detach(libxl__gc *gc, libxl__spawn_state *ss)
+{
+ ss->detaching = 1;
+ spawn_detach(gc, ss);
}
-static void spawn_failed(libxl__egc *egc, libxl__spawn_state *ss)
+static void spawn_fail(libxl__egc *egc, libxl__spawn_state *ss)
+/* Caller must have logged. Must be last thing in calling function,
+ * as it may make the callback. Precondition: Attached or Detaching. */
{
EGC_GC;
- spawn_cleanup(gc, ss);
- ss->failure_cb(egc, ss); /* must be last; callback may do anything to ss
*/
+ ss->failed = 1;
+ spawn_detach(gc, ss);
}
static void spawn_timeout(libxl__egc *egc, libxl__ev_time *ev,
const struct timeval *requested_abs)
{
- /* Before event, was Active; is now Partial. */
+ /* Before event, was Attached. */
EGC_GC;
libxl__spawn_state *ss = CONTAINER_OF(ev, *ss, timeout);
LOG(ERROR, "%s: startup timed out", ss->what);
- spawn_failed(egc, ss); /* must be last */
+ spawn_fail(egc, ss); /* must be last */
}
static void spawn_watch_event(libxl__egc *egc, libxl__ev_xswatch *xsw,
const char *watch_path, const char *event_path)
{
- /* On entry, is Active. */
+ /* On entry, is Attached. */
EGC_GC;
libxl__spawn_state *ss = CONTAINER_OF(xsw, *ss, xswatch);
char *p = libxl__xs_read(gc, 0, ss->xspath);
if (!p && errno != ENOENT) {
LOG(ERROR, "%s: xenstore read of %s failed", ss->what,
ss->xspath);
- spawn_failed(egc, ss); /* must be last */
+ spawn_fail(egc, ss); /* must be last */
return;
}
ss->confirm_cb(egc, ss, p); /* must be last */
@@ -399,20 +408,22 @@ static void spawn_watch_event(libxl__egc *egc,
libxl__ev_xswatch *xsw,
static void spawn_middle_death(libxl__egc *egc, libxl__ev_child *childw,
pid_t pid, int status)
- /* Before event, was Active or Detached;
- * is now Active or Detached except that ssd->mid is Idle */
+ /* On entry, is Attached or Detaching */
{
EGC_GC;
- libxl__spawn_state_detachable *ssd = CONTAINER_OF(childw, *ssd, mid);
- libxl__spawn_state *ss = ssd->ss;
-
- if (!WIFEXITED(status)) {
+ libxl__spawn_state *ss = CONTAINER_OF(childw, *ss, mid);
+
+ if ((ss->failed || ss->detaching) &&
+ ((WIFEXITED(status) && WEXITSTATUS(status)==0) ||
+ (WIFSIGNALED(status) && WTERMSIG(status)==SIGKILL))) {
+ /* as expected */
+ } else if (!WIFEXITED(status)) {
+ int loglevel = ss->detaching ? XTL_WARN : XTL_ERROR;
const char *what - GCSPRINTF("%s intermediate process
(startup monitor)",
- ss ? ss->what : "(detached)");
- int loglevel = ss ? XTL_ERROR : XTL_WARN;
+ GCSPRINTF("%s intermediate process (startup monitor)",
ss->what);
libxl_report_child_exitstatus(CTX, loglevel, what, pid, status);
- } else if (ss) { /* otherwise it was supposed to be a daemon by now */
+ ss->failed = 1;
+ } else {
if (!status)
LOG(ERROR, "%s [%ld]: unexpectedly exited with exit status
0,"
" when we were waiting for it to confirm startup",
@@ -430,15 +441,22 @@ static void spawn_middle_death(libxl__egc *egc,
libxl__ev_child *childw,
LOG(ERROR, "%s [%ld]: died during startup due to unknown
fatal"
" signal number %d", ss->what, (unsigned
long)pid, sig);
}
- ss->ssd = 0; /* detatch the ssd to make the ss be in state Partial
*/
- spawn_failed(egc, ss); /* must be last use of ss */
+ ss->failed = 1;
}
- free(ssd);
-}
-void libxl__spawn_detach(libxl__gc *gc, libxl__spawn_state *ss)
-{
spawn_cleanup(gc, ss);
+
+ if (ss->failed && !ss->detaching) {
+ ss->failure_cb(egc, ss); /* must be last */
+ return;
+ }
+
+ if (ss->failed && ss->detaching)
+ LOG(WARN,"%s underlying machinery seemed to fail,"
+ " but its function seems to have been successful",
ss->what);
+
+ assert(ss->detaching);
+ ss->detached_cb(egc, ss);
}
/*
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8635764..f9ee949 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -690,8 +690,7 @@ static inline int libxl__ev_xswatch_isregistered(const
libxl__ev_xswatch *xw)
* the libxl event machinery.
*
* The parent may signal the child but it must not reap it. That will
- * be done by the event machinery. death may be NULL in which case
- * the child is still reaped but its death is ignored.
+ * be done by the event machinery.
*
* It is not possible to "deregister" the child death event source.
* It will generate exactly one event callback; until then the childw
@@ -998,8 +997,8 @@ _hidden int libxl__device_pci_destroy_all(libxl__gc *gc,
uint32_t domid);
*
* Higher-level double-fork and separate detach eg as for device models
*
- * Each libxl__spawn_state is in one of the conventional states
- * Undefined, Idle, Active
+ * Each libxl__spawn_state is in one of these states
+ * Undefined, Idle, Attached, Detaching
*/
typedef struct libxl__obsolete_spawn_starting libxl__spawn_starting;
@@ -1040,15 +1039,15 @@ _hidden void libxl__spawn_init(libxl__spawn_state*);
* intermediate or final child; an error message will have been
* logged.
*
- * confirm_cb and failure_cb will not be called reentrantly from
- * within libxl__spawn_spawn.
+ * confirm_cb, failure_cb and detached_cb will not be called
+ * reentrantly from within libxl__spawn_spawn.
*
* what: string describing the spawned process, used for logging
*
* Logs errors. A copy of "what" is taken.
* Return values:
* < 0 error, *spawn is now Idle and need not be detached
- * +1 caller is the parent, *spawn is Active and must eventually be
detached
+ * +1 caller is the parent, *spawn is Attached and must be detached
* 0 caller is now the inner child, should probably call libxl__exec
*
* The spawn state must be Undefined or Idle on entry.
@@ -1056,12 +1055,15 @@ _hidden void libxl__spawn_init(libxl__spawn_state*);
_hidden int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *spawn);
/*
- * libxl__spawn_detach - Detaches the daemonic child.
+ * libxl__spawn_request_detach - Detaches the daemonic child.
*
* Works by killing the intermediate process from spawn_spawn.
* After this function returns, failures of either child are no
* longer reported via failure_cb.
*
+ * This is not synchronous: there will be a further callback when
+ * the detach is complete.
+ *
* If called before the inner child has been created, this may prevent
* it from running at all. Thus this should be called only when the
* inner child has notified that it is ready. Normally it will be
@@ -1069,10 +1071,10 @@ _hidden int libxl__spawn_spawn(libxl__egc *egc,
libxl__spawn_state *spawn);
*
* Logs errors.
*
- * The spawn state must be Active or Idle on entry and will be Idle
+ * The spawn state must be Attached entry and will be Detaching
* on return.
*/
-_hidden void libxl__spawn_detach(libxl__gc *gc, libxl__spawn_state*);
+_hidden void libxl__spawn_initiate_detach(libxl__gc *gc, libxl__spawn_state*);
/*
* If successful, this should return 0.
@@ -1109,15 +1111,11 @@ typedef void libxl__spawn_failure_cb(libxl__egc*,
libxl__spawn_state*);
typedef void libxl__spawn_confirm_cb(libxl__egc*, libxl__spawn_state*,
const char *xsdata);
-typedef struct {
- /* Private to the spawn implementation.
- */
- /* This separate struct, from malloc, allows us to "detach"
- * the child and reap it later, when our user has gone
- * away and freed its libxl__spawn_state */
- struct libxl__spawn_state *ss;
- libxl__ev_child mid;
-} libxl__spawn_state_detachable;
+/*
+ * Called when the detach (requested by libxl__spawn_initiate_detach) has
+ * completed. On entry to the callback the spawn state is Idle.
+ */
+typedef void libxl__spawn_detached_cb(libxl__egc*, libxl__spawn_state*);
struct libxl__spawn_state {
/* must be filled in by user and remain valid */
@@ -1129,15 +1127,18 @@ struct libxl__spawn_state {
libxl__spawn_midproc_cb *midproc_cb;
libxl__spawn_failure_cb *failure_cb;
libxl__spawn_confirm_cb *confirm_cb;
+ libxl__spawn_detached_cb *detached_cb;
/* remaining fields are private to libxl_spawn_... */
+ int detaching; /* we are in Detaching */
+ int failed; /* might be true whenever we are not Idle */
+ libxl__ev_child mid; /* always in use whenever we are not Idle */
libxl__ev_time timeout;
libxl__ev_xswatch xswatch;
- libxl__spawn_state_detachable *ssd;
};
static inline int libxl__spawn_inuse(libxl__spawn_state *ss)
- { return !!ss->ssd; }
+ { return libxl__ev_child_inuse(&ss->mid); }
/*
* libxl_spawner_record_pid - Record given pid in xenstore
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 18/19] libxl: do not leak an event struct on ignored ao progress
On entry to libxl__ao_progress_report, the caller has allocated an
event. If the progress report is to be ignored, we need to free it.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/libxl_event.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index eb23a93..1957505 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -1602,6 +1602,7 @@ void libxl__ao_progress_report(libxl__egc *egc, libxl__ao
*ao,
ev->for_user = how->for_event;
if (how->callback == dummy_asyncprogress_callback_ignore) {
LOG(DEBUG,"ao %p: progress report: ignored",ao);
+ libxl_event_free(CTX,ev);
/* ignore */
} else if (how->callback) {
libxl__aop_occurred *aop = libxl__zalloc(&egc->gc,
sizeof(*aop));
--
1.7.2.5
Ian Jackson
2012-Jun-08 17:34 UTC
[PATCH 19/19] libxl: DO NOT APPLY enforce prohibition on internal
DO NOT APPLY THIS PATCH.
It contains -Wno-error. Without that it would break the build.
Subject [PATCH] libxl: enforce prohibitions of internal callers
libxl_internal.h says:
* Functions using LIBXL__INIT_EGC may *not* generally be called from
* within libxl, because libxl__egc_cleanup may call back into the
* application. ...
and
* ... [Functions which take an ao_how] MAY NOT
* be called from inside libxl, because they can cause reentrancy
* callbacks.
However, this was not enforced. Particularly the latter restriction
is easy to overlook, especially since during the transition period to
the new event system we have bent this rule a couple of times, and the
bad pattern simply involves passing 0 or NULL for the ao_how.
So use the compiler to enforce this property, as follows:
- Mark all functions which take a libxl_asyncop_how, or which
use EGC_INIT or LIBXL__INIT_EGC, with a new annotation
LIBXL_EXTERNAL_CALLERS_ONLY in the public header.
- Change the documentation comment for asynch operations and egcs to
say that this should always be done.
- Arrange that if libxl.h is included via libxl_internal.h,
LIBXL_EXTERNAL_CALLERS_ONLY expands to __attribute__((warning(...))),
which generates a message like this:
libxl.c:1772: warning: call to
''libxl_device_disk_remove''
declared with attribute warning:
may not be called from within libxl
Otherwise, the annotation expands to nothing, so external
callers are unaffected.
- Forbid inclusion of both libxl.h and libxl_internal.h unless
libxl_internal.h came first, so that the above check doesn''t have
any loopholes. Files which include libxl_internal.h should not
include libxl.h as well.
This is enforced explicitly using #error. However, in practice
with the current tree it just changes the error message when this
mistake is made; otherwise we would carry on to immediately
following #define which would cause the compiler to complain that
LIBXL_EXTERNAL_CALLERS_ONLY was redefined. Then the developer
might be tempted to add a #ifndef which would be wrong - it would
leave the affected translation unit unprotected by the new
enforcement regime. So let''s be explicit.
- Fix the one source of files which violate the above principle, the
output from the idl compiler, by removing the redundant inclusion
of libxl.h from the output.
In the tree I am using as a base at the time of writing, this new
restriction catches three errors: two in libxl_device_disk_remove and
one in libxl__device_disk_local_detach. To avoid entirely breaking my
build I have also done this:
- Temporarily change -Werror to -Wno-error in the libxl Makefile.
This patch should not be applied in this form.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
---
tools/libxl/Makefile | 2 +-
tools/libxl/gentypes.py | 1 -
tools/libxl/libxl.h | 34 +++++++++++++++++++++++++---------
tools/libxl/libxl_event.h | 21 ++++++++++++++-------
tools/libxl/libxl_internal.h | 14 ++++++++++----
5 files changed, 50 insertions(+), 22 deletions(-)
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 1b81963..deff6f8 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -11,7 +11,7 @@ MINOR = 0
XLUMAJOR = 1.0
XLUMINOR = 0
-CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \
+CFLAGS += -Wno-error -Wno-format-zero-length -Wmissing-declarations \
-Wno-declaration-after-statement -Wformat-nonliteral
CFLAGS += -I. -fPIC
diff --git a/tools/libxl/gentypes.py b/tools/libxl/gentypes.py
index 3c561ba..6e83b21 100644
--- a/tools/libxl/gentypes.py
+++ b/tools/libxl/gentypes.py
@@ -341,7 +341,6 @@ if __name__ == ''__main__'':
#include <stdlib.h>
#include <string.h>
-#include "libxl.h"
#include "libxl_internal.h"
#define LIBXL_DTOR_POISON 0xa5
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 10d7115..1a32d9e 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -266,6 +266,13 @@
#endif
#endif
+/* Functions annotated with LIBXL_EXTERNAL_CALLERS_ONLY may not be
+ * called from within libxl itself. Callers outside libxl, who
+ * do not #include libxl_internal.h, are fine. */
+#ifndef LIBXL_EXTERNAL_CALLERS_ONLY
+#define LIBXL_EXTERNAL_CALLERS_ONLY /* disappears for callers outside libxl */
+#endif
+
typedef uint8_t libxl_mac[6];
#define LIBXL_MAC_FMT "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx"
#define LIBXL_MAC_FMTLEN ((2*6)+5) /* 6 hex bytes plus 5 colons */
@@ -495,11 +502,13 @@ int libxl_ctx_free(libxl_ctx *ctx /* 0 is OK */);
int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid,
const libxl_asyncop_how *ao_how,
- const libxl_asyncprogress_how *aop_console_how);
+ const libxl_asyncprogress_how *aop_console_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd,
const libxl_asyncop_how *ao_how,
- const libxl_asyncprogress_how
*aop_console_how);
+ const libxl_asyncprogress_how *aop_console_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* A progress report will be made via ao_console_how, of type
* domain_create_console_available, when the domain''s primary
* console is available and can be connected to.
@@ -510,7 +519,8 @@ void libxl_domain_config_dispose(libxl_domain_config
*d_config);
int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd,
int flags, /* LIBXL_SUSPEND_* */
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
#define LIBXL_SUSPEND_DEBUG 1
#define LIBXL_SUSPEND_LIVE 2
@@ -522,7 +532,8 @@ int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int
suspend_cancel);
int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
uint32_t domid, int send_fd, int recv_fd,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_shutdown(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_reboot(libxl_ctx *ctx, uint32_t domid);
@@ -544,7 +555,8 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid);
int libxl_domain_core_dump(libxl_ctx *ctx, uint32_t domid,
const char *filename,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint32_t
target_memkb);
int libxl_set_memory_target(libxl_ctx *ctx, uint32_t domid, int32_t
target_memkb, int relative, int enforce);
@@ -653,7 +665,8 @@ void libxl_vminfo_list_free(libxl_vminfo *list, int nr);
int libxl_device_disk_add(libxl_ctx *ctx, uint32_t domid, libxl_device_disk
*disk);
int libxl_device_disk_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_disk_destroy(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk);
@@ -671,7 +684,8 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid,
libxl_device_disk *disk);
int libxl_device_nic_add(libxl_ctx *ctx, uint32_t domid, libxl_device_nic
*nic);
int libxl_device_nic_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_nic *nic,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_nic_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_nic
*nic);
libxl_device_nic *libxl_device_nic_list(libxl_ctx *ctx, uint32_t domid, int
*num);
@@ -682,14 +696,16 @@ int libxl_device_nic_getinfo(libxl_ctx *ctx, uint32_t
domid,
int libxl_device_vkb_add(libxl_ctx *ctx, uint32_t domid, libxl_device_vkb
*vkb);
int libxl_device_vkb_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_vkb *vkb,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_vkb_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_vkb
*vkb);
/* Framebuffer */
int libxl_device_vfb_add(libxl_ctx *ctx, uint32_t domid, libxl_device_vfb
*vfb);
int libxl_device_vfb_remove(libxl_ctx *ctx, uint32_t domid,
libxl_device_vfb *vfb,
- const libxl_asyncop_how *ao_how);
+ const libxl_asyncop_how *ao_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_device_vfb_destroy(libxl_ctx *ctx, uint32_t domid, libxl_device_vfb
*vfb);
/* PCI Passthrough */
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index 713d96d..3344bc8 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -37,7 +37,8 @@ typedef int libxl_event_predicate(const libxl_event*, void
*user);
int libxl_event_check(libxl_ctx *ctx, libxl_event **event_r,
uint64_t typemask,
- libxl_event_predicate *predicate, void *predicate_user);
+ libxl_event_predicate *predicate, void *predicate_user)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* Searches for an event, already-happened, which matches typemask
* and predicate. predicate==0 matches any event.
* libxl_event_check returns the event, which must then later be
@@ -48,7 +49,8 @@ int libxl_event_check(libxl_ctx *ctx, libxl_event **event_r,
int libxl_event_wait(libxl_ctx *ctx, libxl_event **event_r,
uint64_t typemask,
- libxl_event_predicate *predicate, void *predicate_user);
+ libxl_event_predicate *predicate, void *predicate_user)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* Like libxl_event_check but blocks if no suitable events are
* available, until some are. Uses libxl_osevent_beforepoll/
* _afterpoll so may be inefficient if very many domains are being
@@ -256,7 +258,8 @@ struct pollfd;
*/
int libxl_osevent_beforepoll(libxl_ctx *ctx, int *nfds_io,
struct pollfd *fds, int *timeout_upd,
- struct timeval now);
+ struct timeval now)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* nfds and fds[0..nfds] must be from the most recent call to
* _beforepoll, as modified by poll. (It is therefore not possible
@@ -271,7 +274,8 @@ int libxl_osevent_beforepoll(libxl_ctx *ctx, int *nfds_io,
* libxl_event_check.
*/
void libxl_osevent_afterpoll(libxl_ctx *ctx, int nfds, const struct pollfd
*fds,
- struct timeval now);
+ struct timeval now)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
typedef struct libxl_osevent_hooks {
@@ -357,14 +361,16 @@ void libxl_osevent_register_hooks(libxl_ctx *ctx,
*/
void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
- int fd, short events, short revents);
+ int fd, short events, short revents)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/* Implicitly, on entry to this function the timeout has been
* deregistered. If _occurred_timeout is called, libxl will not
* call timeout_deregister; if it wants to requeue the timeout it
* will call timeout_register again.
*/
-void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
+void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/*======================================================================*/
@@ -506,7 +512,8 @@ void libxl_childproc_setmode(libxl_ctx *ctx, const
libxl_childproc_hooks *hooks,
* certainly need to use the self-pipe trick (or a working pselect or
* ppoll) to implement this.
*/
-int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status);
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status)
+ LIBXL_EXTERNAL_CALLERS_ONLY;
/*
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index f9ee949..0972f0e 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -52,6 +52,12 @@
#include <xen/io/xenbus.h>
+#ifdef LIBXL_H
+# error libxl.h should be included via libxl_internal.h, not separately
+#endif
+#define LIBXL_EXTERNAL_CALLERS_ONLY \
+ __attribute__((warning("may not be called from within libxl")))
+
#include "libxl.h"
#include "_paths.h"
#include "_libxl_save_msgs_callout.h"
@@ -1537,10 +1543,10 @@ libxl__device_model_version_running(libxl__gc *gc,
uint32_t domid);
*
* Functions using LIBXL__INIT_EGC may *not* generally be called from
* within libxl, because libxl__egc_cleanup may call back into the
- * application. This should be documented near the function
- * prototype(s) for callers of LIBXL__INIT_EGC and EGC_INIT. You
- * should in any case not find it necessary to call egc-creators from
- * within libxl.
+ * application. This should be enforced by declaring all such
+ * functions in libxl.h or libxl_event.h with
+ * LIBXL_EXTERNAL_CALLERS_ONLY. You should in any case not find it
+ * necessary to call egc-creators from within libxl.
*
* The callbacks must all take place with the ctx unlocked because
* the application is entitled to reenter libxl from them. This
--
1.7.2.5
Ian Jackson
2012-Jun-11 16:43 UTC
[PATCH] libxl: further fixups re LIBXL_DOMAIN_TYPE process
Here''s another patch which needs to go on top of my async save/restore
series.
Ian.
From: Ian Jackson <ian.jackson@eu.citrix.com>
Subject: [PATCH] libxl: further fixups re LIBXL_DOMAIN_TYPE
* Abolish the macro LIBXL__DOMAIN_IS_TYPE which had incorrect error
handlng. At every call site, replace it with an open-coded call to
libxl_domain_type and check against LIBXL_DOMAIN_TYPE_INVALID.
* This involves adding an `out:'' to libxl_domain_unpause.
* In libxl_domain_destroy and do_pci_add, do not `default: abort();''
if the domain type cannot be found. Instead switch on
LIBXL_DOMAIN_TYPE_INVALID specifically and do some actual error
handling.
* In libxl__primary_console_find, remove a spurious default clause
from the domain type switch.
* In libxl_domain_suspend (as reorganised) error check, check for
LIBXL_DOMAIN_TYPE_INVALID and remove a pointless extra log message.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tools/libxl/libxl.c | 33 ++++++++++++++++++++++++---------
tools/libxl/libxl_internal.h | 5 +++--
tools/libxl/libxl_pci.c | 18 +++++++++++++-----
3 files changed, 40 insertions(+), 16 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index a257902..cd2dbda 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -390,7 +390,13 @@ int libxl_domain_resume(libxl_ctx *ctx, uint32_t domid, int
suspend_cancel)
goto out;
}
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, HVM)) {
+ libxl_domain_type type = libxl__domain_type(gc, domid);
+ if (type == LIBXL_DOMAIN_TYPE_INVALID) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (type == LIBXL_DOMAIN_TYPE_HVM) {
rc = libxl__domain_resume_device_model(gc, domid);
if (rc) {
LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
@@ -723,8 +729,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int
fd, int flags,
int rc;
libxl_domain_type type = libxl__domain_type(gc, domid);
- if (type < 0) {
- LOG(ERROR,"domain %"PRIu32": unable to determine domain
type", domid);
+ if (type == LIBXL_DOMAIN_TYPE_INVALID) {
rc = ERROR_FAIL;
goto out_err;
}
@@ -789,7 +794,13 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid)
char *state;
int ret, rc = 0;
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, HVM)) {
+ libxl_domain_type type = libxl__domain_type(gc, domid);
+ if (type == LIBXL_DOMAIN_TYPE_INVALID) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (type == LIBXL_DOMAIN_TYPE_HVM) {
path = libxl__sprintf(gc,
"/local/domain/0/device-model/%d/state", domid);
state = libxl__xs_read(gc, XBT_NULL, path);
if (state != NULL && !strcmp(state, "paused")) {
@@ -803,6 +814,7 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid)
LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unpausing domain
%d", domid);
rc = ERROR_FAIL;
}
+ out:
GC_FREE;
return rc;
}
@@ -814,7 +826,11 @@ int libxl__domain_pvcontrol_available(libxl__gc *gc,
uint32_t domid)
unsigned long pvdriver = 0;
int ret;
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV))
+ libxl_domain_type domtype = libxl__domain_type(gc, domid);
+ if (domtype == LIBXL_DOMAIN_TYPE_INVALID)
+ return ERROR_FAIL;
+
+ if (domtype == LIBXL_DOMAIN_TYPE_PV)
return 1;
ret = xc_get_hvm_param(ctx->xch, domid, HVM_PARAM_CALLBACK_IRQ,
&pvdriver);
@@ -1214,8 +1230,9 @@ int libxl_domain_destroy(libxl_ctx *ctx, uint32_t domid)
pid = libxl__xs_read(gc, XBT_NULL, libxl__sprintf(gc,
"/local/domain/%d/image/device-model-pid", domid));
dm_present = (pid != NULL);
break;
- default:
- abort();
+ case LIBXL_DOMAIN_TYPE_INVALID:
+ rc = ERROR_FAIL;
+ goto out;
}
dom_path = libxl__xs_get_dompath(gc, domid);
@@ -1363,8 +1380,6 @@ static int libxl__primary_console_find(libxl_ctx *ctx,
uint32_t domid_vm,
case LIBXL_DOMAIN_TYPE_INVALID:
rc = ERROR_INVAL;
goto out;
- default:
- abort();
}
}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index f9ee949..c9725a3 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -797,8 +797,7 @@ _hidden int libxl__domain_cpupool(libxl__gc *gc, uint32_t
domid);
_hidden libxl_scheduler libxl__domain_scheduler(libxl__gc *gc, uint32_t domid);
_hidden int libxl__sched_set_params(libxl__gc *gc, uint32_t domid,
libxl_domain_sched_params *scparams);
-#define LIBXL__DOMAIN_IS_TYPE(gc, domid, type) \
- libxl__domain_type((gc), (domid)) == LIBXL_DOMAIN_TYPE_##type
+
typedef struct {
uint32_t store_port;
uint32_t store_domid;
@@ -841,7 +840,9 @@ _hidden int libxl__domain_resume_device_model(libxl__gc *gc,
uint32_t domid);
_hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid);
+/* returns 0 or 1, or a libxl error code */
_hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid);
+
_hidden char * libxl__domain_pvcontrol_read(libxl__gc *gc,
xs_transaction_t t, uint32_t
domid);
_hidden int libxl__domain_pvcontrol_write(libxl__gc *gc, xs_transaction_t t,
diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c
index de1b79f..81438be 100644
--- a/tools/libxl/libxl_pci.c
+++ b/tools/libxl/libxl_pci.c
@@ -128,7 +128,11 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc,
uint32_t domid, libxl_d
if (!num_devs)
return libxl__create_pci_backend(gc, domid, pcidev, 1);
- if (!starting && LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) {
+ libxl_domain_type domtype = libxl__domain_type(gc, domid);
+ if (domtype == LIBXL_DOMAIN_TYPE_INVALID)
+ return ERROR_FAIL;
+
+ if (!starting && domtype == LIBXL_DOMAIN_TYPE_PV) {
if (libxl__wait_for_backend(gc, be_path, "4") < 0)
return ERROR_FAIL;
}
@@ -171,7 +175,11 @@ static int libxl__device_pci_remove_xenstore(libxl__gc *gc,
uint32_t domid, libx
return ERROR_INVAL;
num = atoi(num_devs);
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) {
+ libxl_domain_type domtype = libxl__domain_type(gc, domid);
+ if (domtype == LIBXL_DOMAIN_TYPE_INVALID)
+ return ERROR_FAIL;
+
+ if (domtype == LIBXL_DOMAIN_TYPE_PV) {
if (libxl__wait_for_backend(gc, be_path, "4") < 0) {
LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not
ready", be_path);
return ERROR_FAIL;
@@ -199,7 +207,7 @@ retry_transaction:
if (errno == EAGAIN)
goto retry_transaction;
- if (LIBXL__DOMAIN_IS_TYPE(gc, domid, PV)) {
+ if (domtype == LIBXL_DOMAIN_TYPE_PV) {
if (libxl__wait_for_backend(gc, be_path, "4") < 0) {
LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not
ready", be_path);
return ERROR_FAIL;
@@ -939,8 +947,8 @@ static int do_pci_add(libxl__gc *gc, uint32_t domid,
libxl_device_pci *pcidev, i
}
break;
}
- default:
- abort();
+ case LIBXL_DOMAIN_TYPE_INVALID:
+ return ERROR_FAIL;
}
out:
if (!libxl_is_stubdom(ctx, domid, NULL)) {
--
tg: (966769c..) t/xen/xl.domain-type-fixups-v2 (depends on:
t/xen/xl.ao.progress-ignored-free-event)
Ian Campbell
2012-Jun-12 15:15 UTC
Re: [PATCH 01/19] libxc: xc_domain_restore, make toolstack_restore const-correct
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> Update the one provider of this callback, in libxl. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com>> Changes in v3: > * No longer introduce function pointer typedefs into the libxc API.Thanks!> --- > tools/libxc/xenguest.h | 2 +- > tools/libxl/libxl_dom.c | 4 ++-- > 2 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/tools/libxc/xenguest.h b/tools/libxc/xenguest.h > index 91d53f7..707e31c 100644 > --- a/tools/libxc/xenguest.h > +++ b/tools/libxc/xenguest.h > @@ -92,7 +92,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter > /* callbacks provided by xc_domain_restore */ > struct restore_callbacks { > /* callback to restore toolstack specific data */ > - int (*toolstack_restore)(uint32_t domid, uint8_t *buf, > + int (*toolstack_restore)(uint32_t domid, const uint8_t *buf, > uint32_t size, void* data); > > /* to be provided as the last argument to each callback function */ > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index 10f8c1f..677db1d 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -467,13 +467,13 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t domid, > domid, phys_offset, node); > } > > -static int libxl__toolstack_restore(uint32_t domid, uint8_t *buf, > +static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > uint32_t size, void *data) > { > libxl__gc *gc = (libxl__gc *) data; > libxl_ctx *ctx = gc->owner; > int i, ret; > - uint8_t *ptr = buf; > + const uint8_t *ptr = buf; > uint32_t count = 0, version = 0; > struct libxl__physmap_info* pi; > char *xs_path;
Ian Campbell
2012-Jun-12 15:24 UTC
Re: [PATCH 02/19] libxl: domain save: rename variables etc.
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> Preparatory work for making domain suspend asynchronous:[...]> No functional change whatsoever.Given that and a quick skim of the changes: Acked-by: Ian Campbell <ian.campbell@citrix.com> Although in general I find it easier to review patches which contain only one particular class of "mostly automatic" cleanup (e.g. renames separate from "pointerization" separate from local variable removal etc). Ian.
Ian Campbell
2012-Jun-12 15:49 UTC
Re: [PATCH 03/19] libxl: domain restore: reshuffle, preparing for ao
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> We are going to arrange that libxl, instead of calling > xc_domain_restore, calls a stub function which forks and execs a > helper program, so that restore can be asynchronous rather than > blocking the whole toolstack. > > This stub function will be called libxl__xc_domain_restore. > > However, its prospective call site is unsuitable for a function which > needs to make a callback, and is buried in two nested single-call-site > functions which are logically part of the domain creation procedure. > > So we first abolish those single-call-site functions, integrate their > contents into domain creation in their proper temporal order, and > break out libxl__xc_domain_restore ready for its reimplementation. > > No functional change - just the following reorganisation: > > * Abolish libxl__domain_restore_common, as it had only one caller. > Move its contents into (what was) domain_restore. > > * There is a new stage function domcreate_rebuild_done containing what > used to be the bulk of domcreate_bootloader_done, since > domcreate_bootloader_done now simply starts the restore (or does the > rebuild) and arranges to call the next stage. > > * Move the contents of domain_restore into its correct place in the > domain creation sequence. We put it inside > domcreate_bootloader_done, which now either calls"either" but no "or"? I suppose the or case is call domcreate_rebuild_done directly?> libxl__xc_domain_restore which will call the new function > domcreate_rebuild_done. > > * Various general-purpose local variables (`i'' etc.) and convenience > alias variables need to be shuffled about accordingly. > > * Consequently libxl__toolstack_restore needs to gain external linkage > as it is now in a different file to its user. > > * Move the xc_domain_save callbacks struct from the stack into > libxl__domain_create_state. > > In general the moved code remains almost identical. Two returns in > what used to be libxl__domain_restore_common have been changed to set > the return value and "goto out", and the call sites for the abolished > and new functions have been adjusted. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com> I confirmed at a high level that the same blocks of code exist and are called in the same overall order, but I didn''t do a line by line comparison of the blocks in question, assuming they really are mostly motion and adjustments for the new context.> > Changes in v2: > * Also move the save callbacks > --- > tools/libxl/Makefile | 1 + > tools/libxl/libxl_create.c | 244 +++++++++++++++++++++++-------------- > tools/libxl/libxl_dom.c | 45 +------- > tools/libxl/libxl_internal.h | 19 +++- > tools/libxl/libxl_save_callout.c | 37 ++++++ > 5 files changed, 206 insertions(+), 140 deletions(-) > create mode 100644 tools/libxl/libxl_save_callout.c > > diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile > index e7d5cc2..1d8b80a 100644 > --- a/tools/libxl/Makefile > +++ b/tools/libxl/Makefile > @@ -67,6 +67,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \ > libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \ > libxl_internal.o libxl_utils.o libxl_uuid.o \ > libxl_json.o libxl_aoutils.o \ > + libxl_save_callout.o \ > libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y) > LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o > > diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c > index 4456ae8..4439367 100644 > --- a/tools/libxl/libxl_create.c > +++ b/tools/libxl/libxl_create.c > @@ -21,7 +21,6 @@ > #include "libxl_arch.h" > > #include <xc_dom.h> > -#include <xenguest.h> > > void libxl_domain_config_init(libxl_domain_config *d_config) > { > @@ -317,89 +316,6 @@ out: > return ret; > } > > -static int domain_restore(libxl__gc *gc, libxl_domain_build_info *info, > - uint32_t domid, int fd, > - libxl__domain_build_state *state) > -{ > - libxl_ctx *ctx = libxl__gc_owner(gc); > - char **vments = NULL, **localents = NULL; > - struct timeval start_time; > - int i, ret, esave, flags; > - > - ret = libxl__build_pre(gc, domid, info, state); > - if (ret) > - goto out; > - > - ret = libxl__domain_restore_common(gc, domid, info, state, fd); > - if (ret) > - goto out; > - > - gettimeofday(&start_time, NULL); > - > - switch (info->type) { > - case LIBXL_DOMAIN_TYPE_HVM: > - vments = libxl__calloc(gc, 7, sizeof(char *)); > - vments[0] = "rtc/timeoffset"; > - vments[1] = (info->u.hvm.timeoffset) ? info->u.hvm.timeoffset : ""; > - vments[2] = "image/ostype"; > - vments[3] = "hvm"; > - vments[4] = "start_time"; > - vments[5] = libxl__sprintf(gc, "%lu.%02d", start_time.tv_sec,(int)start_time.tv_usec/10000); > - break; > - case LIBXL_DOMAIN_TYPE_PV: > - vments = libxl__calloc(gc, 11, sizeof(char *)); > - i = 0; > - vments[i++] = "image/ostype"; > - vments[i++] = "linux"; > - vments[i++] = "image/kernel"; > - vments[i++] = (char *) state->pv_kernel.path; > - vments[i++] = "start_time"; > - vments[i++] = libxl__sprintf(gc, "%lu.%02d", start_time.tv_sec,(int)start_time.tv_usec/10000); > - if (state->pv_ramdisk.path) { > - vments[i++] = "image/ramdisk"; > - vments[i++] = (char *) state->pv_ramdisk.path; > - } > - if (state->pv_cmdline) { > - vments[i++] = "image/cmdline"; > - vments[i++] = (char *) state->pv_cmdline; > - } > - break; > - default: > - ret = ERROR_INVAL; > - goto out; > - } > - ret = libxl__build_post(gc, domid, info, state, vments, localents); > - if (ret) > - goto out; > - > - if (info->type == LIBXL_DOMAIN_TYPE_HVM) { > - ret = asprintf(&state->saved_state, > - XC_DEVICE_MODEL_RESTORE_FILE".%d", domid); > - ret = (ret < 0) ? ERROR_FAIL : 0; > - } > - > -out: > - if (info->type == LIBXL_DOMAIN_TYPE_PV) { > - libxl__file_reference_unmap(&state->pv_kernel); > - libxl__file_reference_unmap(&state->pv_ramdisk); > - } > - > - esave = errno; > - > - flags = fcntl(fd, F_GETFL); > - if (flags == -1) { > - LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to get flags on restore fd"); > - } else { > - flags &= ~O_NONBLOCK; > - if (fcntl(fd, F_SETFL, flags) == -1) > - LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to put restore fd" > - " back to blocking mode"); > - } > - > - errno = esave; > - return ret; > -} > - > int libxl__domain_make(libxl__gc *gc, libxl_domain_create_info *info, > uint32_t *domid) > { > @@ -580,10 +496,13 @@ static void domcreate_bootloader_console_available(libxl__egc *egc, > static void domcreate_bootloader_done(libxl__egc *egc, > libxl__bootloader_state *bl, > int rc); > - > static void domcreate_console_available(libxl__egc *egc, > libxl__domain_create_state *dcs); > > +static void domcreate_rebuild_done(libxl__egc *egc, > + libxl__domain_create_state *dcs, > + int ret); > + > /* Our own function to clean up and call the user''s callback. > * The final call in the sequence. */ > static void domcreate_complete(libxl__egc *egc, > @@ -671,20 +590,20 @@ static void domcreate_console_available(libxl__egc *egc, > > static void domcreate_bootloader_done(libxl__egc *egc, > libxl__bootloader_state *bl, > - int ret) > + int rc) > { > libxl__domain_create_state *dcs = CONTAINER_OF(bl, *dcs, bl); > STATE_AO_GC(bl->ao); > - int i; > > /* convenience aliases */ > const uint32_t domid = dcs->guest_domid; > libxl_domain_config *const d_config = dcs->guest_config; > + libxl_domain_build_info *const info = &d_config->b_info; > const int restore_fd = dcs->restore_fd; > libxl__domain_build_state *const state = &dcs->build_state; > - libxl_ctx *const ctx = CTX; > + struct restore_callbacks *const callbacks = &dcs->callbacks; > > - if (ret) goto error_out; > + if (rc) domcreate_rebuild_done(egc, dcs, rc); > > /* consume bootloader outputs. state->pv_{kernel,ramdisk} have > * been initialised by the bootloader already. > @@ -700,12 +619,153 @@ static void domcreate_bootloader_done(libxl__egc *egc, > dcs->dmss.dm.callback = domcreate_devmodel_started; > dcs->dmss.callback = domcreate_devmodel_started; > > - if ( restore_fd >= 0 ) { > - ret = domain_restore(gc, &d_config->b_info, domid, restore_fd, state); > + if ( restore_fd < 0 ) { > + rc = libxl__domain_build(gc, &d_config->b_info, domid, state); > + domcreate_rebuild_done(egc, dcs, rc); > + return; > + } > + > + /* Restore */ > + > + rc = libxl__build_pre(gc, domid, info, state); > + if (rc) > + goto out; > + > + /* read signature */ > + int hvm, pae, superpages; > + int no_incr_generationid; > + switch (info->type) { > + case LIBXL_DOMAIN_TYPE_HVM: > + hvm = 1; > + superpages = 1; > + pae = libxl_defbool_val(info->u.hvm.pae); > + no_incr_generationid = !libxl_defbool_val(info->u.hvm.incr_generationid); > + callbacks->toolstack_restore = libxl__toolstack_restore; > + callbacks->data = gc; > + break; > + case LIBXL_DOMAIN_TYPE_PV: > + hvm = 0; > + superpages = 0; > + pae = 1; > + no_incr_generationid = 0; > + break; > + default: > + rc = ERROR_INVAL; > + goto out; > + } > + libxl__xc_domain_restore(egc, dcs, > + hvm, pae, superpages, no_incr_generationid); > + return; > + > + out: > + libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0); > +} > + > +void libxl__xc_domain_restore_done(libxl__egc *egc, > + libxl__domain_create_state *dcs, > + int ret, int retval, int errnoval) > +{ > + STATE_AO_GC(dcs->ao); > + libxl_ctx *ctx = libxl__gc_owner(gc); > + char **vments = NULL, **localents = NULL; > + struct timeval start_time; > + int i, esave, flags; > + > + /* convenience aliases */ > + const uint32_t domid = dcs->guest_domid; > + libxl_domain_config *const d_config = dcs->guest_config; > + libxl_domain_build_info *const info = &d_config->b_info; > + libxl__domain_build_state *const state = &dcs->build_state; > + const int fd = dcs->restore_fd; > + > + if (ret) > + goto out; > + > + if (retval) { > + LOGEV(ERROR, errnoval, "restoring domain"); > + ret = ERROR_FAIL; > + goto out; > + } > + > + gettimeofday(&start_time, NULL); > + > + switch (info->type) { > + case LIBXL_DOMAIN_TYPE_HVM: > + vments = libxl__calloc(gc, 7, sizeof(char *)); > + vments[0] = "rtc/timeoffset"; > + vments[1] = (info->u.hvm.timeoffset) ? info->u.hvm.timeoffset : ""; > + vments[2] = "image/ostype"; > + vments[3] = "hvm"; > + vments[4] = "start_time"; > + vments[5] = libxl__sprintf(gc, "%lu.%02d", start_time.tv_sec,(int)start_time.tv_usec/10000); > + break; > + case LIBXL_DOMAIN_TYPE_PV: > + vments = libxl__calloc(gc, 11, sizeof(char *)); > + i = 0; > + vments[i++] = "image/ostype"; > + vments[i++] = "linux"; > + vments[i++] = "image/kernel"; > + vments[i++] = (char *) state->pv_kernel.path; > + vments[i++] = "start_time"; > + vments[i++] = libxl__sprintf(gc, "%lu.%02d", start_time.tv_sec,(int)start_time.tv_usec/10000); > + if (state->pv_ramdisk.path) { > + vments[i++] = "image/ramdisk"; > + vments[i++] = (char *) state->pv_ramdisk.path; > + } > + if (state->pv_cmdline) { > + vments[i++] = "image/cmdline"; > + vments[i++] = (char *) state->pv_cmdline; > + } > + break; > + default: > + ret = ERROR_INVAL; > + goto out; > + } > + ret = libxl__build_post(gc, domid, info, state, vments, localents); > + if (ret) > + goto out; > + > + if (info->type == LIBXL_DOMAIN_TYPE_HVM) { > + ret = asprintf(&state->saved_state, > + XC_DEVICE_MODEL_RESTORE_FILE".%d", domid); > + ret = (ret < 0) ? ERROR_FAIL : 0; > + } > + > +out: > + if (info->type == LIBXL_DOMAIN_TYPE_PV) { > + libxl__file_reference_unmap(&state->pv_kernel); > + libxl__file_reference_unmap(&state->pv_ramdisk); > + } > + > + esave = errno; > + > + flags = fcntl(fd, F_GETFL); > + if (flags == -1) { > + LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to get flags on restore fd"); > } else { > - ret = libxl__domain_build(gc, &d_config->b_info, domid, state); > + flags &= ~O_NONBLOCK; > + if (fcntl(fd, F_SETFL, flags) == -1) > + LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unable to put restore fd" > + " back to blocking mode"); > } > > + errno = esave; > + domcreate_rebuild_done(egc, dcs, ret); > +} > + > +static void domcreate_rebuild_done(libxl__egc *egc, > + libxl__domain_create_state *dcs, > + int ret) > +{ > + STATE_AO_GC(dcs->ao); > + int i; > + > + /* convenience aliases */ > + const uint32_t domid = dcs->guest_domid; > + libxl_domain_config *const d_config = dcs->guest_config; > + libxl__domain_build_state *const state = &dcs->build_state; > + libxl_ctx *const ctx = CTX; > + > if (ret) { > LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "cannot (re-)build domain: %d", ret); > ret = ERROR_FAIL; > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index e90030d..1d4e809 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -19,7 +19,6 @@ > > #include <xenctrl.h> > #include <xc_dom.h> > -#include <xenguest.h> > > #include <xen/hvm/hvm_info_table.h> > > @@ -467,7 +466,7 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t domid, > domid, phys_offset, node); > } > > -static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > +int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > uint32_t size, void *data) > { > libxl__gc *gc = data; > @@ -522,48 +521,6 @@ static int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > return 0; > } > > -int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, > - libxl_domain_build_info *info, > - libxl__domain_build_state *state, > - int fd) > -{ > - libxl_ctx *ctx = libxl__gc_owner(gc); > - /* read signature */ > - int rc; > - int hvm, pae, superpages; > - struct restore_callbacks callbacks[1]; > - int no_incr_generationid; > - switch (info->type) { > - case LIBXL_DOMAIN_TYPE_HVM: > - hvm = 1; > - superpages = 1; > - pae = libxl_defbool_val(info->u.hvm.pae); > - no_incr_generationid = !libxl_defbool_val(info->u.hvm.incr_generationid); > - callbacks->toolstack_restore = libxl__toolstack_restore; > - callbacks->data = gc; > - break; > - case LIBXL_DOMAIN_TYPE_PV: > - hvm = 0; > - superpages = 0; > - pae = 1; > - no_incr_generationid = 0; > - break; > - default: > - return ERROR_INVAL; > - } > - rc = xc_domain_restore(ctx->xch, fd, domid, > - state->store_port, &state->store_mfn, > - state->store_domid, state->console_port, > - &state->console_mfn, state->console_domid, > - hvm, pae, superpages, no_incr_generationid, > - &state->vm_generationid_addr, callbacks); > - if ( rc ) { > - LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain"); > - return ERROR_FAIL; > - } > - return 0; > -} > - > static int libxl__domain_suspend_common_switch_qemu_logdirty > (int domid, unsigned int enable, void *data) > { > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index f22bf94..28478ea 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -46,6 +46,7 @@ > > #include <xenstore.h> > #include <xenctrl.h> > +#include <xenguest.h> > > #include "xentoollog.h" > > @@ -782,10 +783,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t domid, > const char *old_name, const char *new_name, > xs_transaction_t trans); > > -_hidden int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, > - libxl_domain_build_info *info, > - libxl__domain_build_state *state, > - int fd); > +_hidden int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > + uint32_t size, void *data); > _hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid); > _hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid); > _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid); > @@ -1899,6 +1898,7 @@ struct libxl__domain_create_state { > libxl__stub_dm_spawn_state dmss; > /* If we''re not doing stubdom, we use only dmss.dm, > * for the non-stubdom device model. */ > + struct restore_callbacks callbacks; > }; > > /*----- Domain suspend (save) functions -----*/ > @@ -1908,6 +1908,17 @@ _hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > int live, int debug, > const libxl_domain_remus_info *r_info); > > +/* calls libxl__xc_domain_restore_done when done */ > +_hidden void libxl__xc_domain_restore(libxl__egc *egc, > + libxl__domain_create_state *dcs, > + int hvm, int pae, int superpages, > + int no_incr_generationid); > +/* If rc==0 then retval is the return value from xc_domain_save > + * and errnoval is the errno value it provided. > + * If rc!=0, retval and errnoval are undefined. */ > +_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, > + libxl__domain_create_state *dcs, > + int rc, int retval, int errnoval); > > > /* > diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c > new file mode 100644 > index 0000000..2f8db9f > --- /dev/null > +++ b/tools/libxl/libxl_save_callout.c > @@ -0,0 +1,37 @@ > +/* > + * Copyright (C) 2012 Citrix Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU Lesser General Public License as published > + * by the Free Software Foundation; version 2.1 only. with the special > + * exception on linking described in file LICENSE. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU Lesser General Public License for more details. > + */ > + > +#include "libxl_osdeps.h" > + > +#include "libxl_internal.h" > + > +void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs, > + int hvm, int pae, int superpages, > + int no_incr_generationid) > +{ > + STATE_AO_GC(dcs->ao); > + > + /* Convenience aliases */ > + const uint32_t domid = dcs->guest_domid; > + const int restore_fd = dcs->restore_fd; > + libxl__domain_build_state *const state = &dcs->build_state; > + > + int r = xc_domain_restore(CTX->xch, restore_fd, domid, > + state->store_port, &state->store_mfn, > + state->store_domid, state->console_port, > + &state->console_mfn, state->console_domid, > + hvm, pae, superpages, no_incr_generationid, > + &state->vm_generationid_addr, &dcs->callbacks); > + libxl__xc_domain_restore_done(egc, dcs, 0, r, errno); > +} > -- > 1.7.2.5 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Ian Campbell
2012-Jun-12 16:51 UTC
Re: [PATCH 04/19] libxl: domain save: API changes for asynchrony
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> Change the internal and external APIs for domain save (suspend) to be > capable of asynchronous operation. The implementation remains > synchronous. The interfaces surrounding device model saving are still > synchronous. > > Public API changes: > > * libxl_domain_save takes an ao_how. > > * libxl_domain_remus_start takes an ao_how. If the > libxl_domain_remus_info is NULL, we abort rather than returning an > error. > > * The `suspend_callback'' function passed to libxl_domain_save is > never called by the existing implementation in libxl. Abolish it.Furthermore xl never passes one in either. [...]> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>A few minor comments below, but otherwise looks good to me. [...]> > +static void remus_crashed_cb(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc)I''m not sure "crashed" is quite right here, it''s finished for whatever reason which may not necessarily be a crash (going forward it should rarely be a crash, I think). It''s "stopped" or "done" or something.> [...]> +int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags, > + const libxl_asyncop_how *ao_how) > +{ > + AO_CREATE(ctx, domid, ao_how); > + int rc; > + > libxl_domain_type type = libxl__domain_type(gc, domid); > - int live = info != NULL && info->flags & XL_SUSPEND_LIVE; > - int debug = info != NULL && info->flags & XL_SUSPEND_DEBUG; > - int rc = 0; > + if (type < 0) { > + LOG(ERROR,"domain %"PRIu32": unable to determine domain type", domid);libxl__domain_type now logs for you.> + rc = ERROR_FAIL; > + goto out_err; > + } > > - rc = libxl__domain_suspend_common(gc, domid, fd, type, live, debug, > - /* No Remus */ NULL); > + libxl__domain_suspend_state *dss; > + GCNEW(dss); > > - if (!rc && type == LIBXL_DOMAIN_TYPE_HVM) > - rc = libxl__domain_save_device_model(gc, domid, fd); > - GC_FREE; > - return rc; > + dss->ao = ao; > + dss->callback = domain_suspend_cb; > + > + dss->domid = domid; > + dss->fd = fd; > + dss->type = type; > + dss->live = flags & LIBXL_SUSPEND_LIVE; > + dss->debug = flags & LIBXL_SUSPEND_DEBUG; > + > + libxl__domain_suspend(egc, dss); > + return AO_INPROGRESS; > + > + out_err: > + return AO_ABORT(rc); > } > > int libxl_domain_pause(libxl_ctx *ctx, uint32_t domid)[...]> @@ -1903,10 +1915,27 @@ struct libxl__domain_create_state { > > /*----- Domain suspend (save) functions -----*/ > > -_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > - libxl_domain_type type, > - int live, int debug, > - const libxl_domain_remus_info *r_info); > +/* calls callback when done */Which callback? dss->callback I guess.> +_hidden void libxl__domain_suspend(libxl__egc *egc, > + libxl__domain_suspend_state *dss); > + > + > +/* calls libxl__xc_domain_suspend_done when done */ > +_hidden void libxl__xc_domain_save(libxl__egc*, libxl__domain_suspend_state*, > + unsigned long vm_generationid_addr); > +/* If rc==0 then retval is the return value from xc_domain_save > + * and errnoval is the errno value it provided. > + * If rc!=0, retval and errnoval are undefined. */ > +_hidden void libxl__xc_domain_save_done(libxl__egc*, > + libxl__domain_suspend_state*, > + int rc, int retval, int errnoval); > + > +_hidden int libxl__domain_suspend_common_callback(void *data); > +_hidden int libxl__domain_suspend_common_switch_qemu_logdirty > + (int domid, unsigned int enable, void *data); > +_hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf, > + uint32_t *len, void *data); > + > > /* calls libxl__xc_domain_restore_done when done */ > _hidden void libxl__xc_domain_restore(libxl__egc *egc,
Ian Campbell
2012-Jun-13 08:59 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> This is v3 of my series to asyncify save/restore, rebased to current > tip, retested, and with all comments addressed.There''s quite a lot of combinations which need testing here (PV, HVM, HVM w/ stub dm, old vs new qemu etc etc), which of those have you tried? I tried a simple localhost migrate of a PV guest and: # xl -vvv migrate d32-1 localhost migration target: Ready to receive domain. Saving to migration stream new xl format (info 0x0/0x0/3541) libxl: debug: libxl.c:722:libxl_domain_suspend: ao 0x8069720: create: how=(nil) callback=(nil) poller=0x80696c8 Loading new save file <incoming migration stream> (new xl fmt info 0x0/0x0/3541) Savefile contains xl domain config libxl: debug: libxl_dom.c:969:libxl__toolstack_save: domain=2 toolstack data size=8 libxl: debug: libxl.c:745:libxl_domain_suspend: ao 0x8069720: inprogress: poller=0x80696c8, flags=i libxl-save-helper: debug: starting save: Success xc: detail: Had 0 unexplained entries in p2m table xc: Saving memory: iter 0 (last sent 0 skipped 0): 0/131072 0% at which point it appears to just stop. # strace -p 2872 # /usr/lib/xen/bin/libxl-save-helper --save-domain 8 2 0 0 1 0 0 12 8 72 Process 2872 attached - interrupt to quit write(8, 0xb5d31000, 1974272^C <unfinished ...> Process 2872 detached # strace -p 2866 # /usr/lib/xen/bin/libxl-save-helper --restore-domain 0 3 1 0 2 0 0 1 0 0 0 Process 2866 attached - interrupt to quit read(0, ^C <unfinished ...> # strace -p 4070 # xl -vvv migrate d32-1 localhost Process 4070 attached - interrupt to quit restart_syscall(<... resuming interrupted call ...> # strace -p 4074 # xl migrate-receive Process 4074 attached - interrupt to quit restart_syscall(<... resuming interrupted call ...> So the saver seems to be blocked writing to fd 8, which is argv[1] == io_fd. Also FWIW: # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 511 4 r----- 24.5 d32-1 2 128 4 -b---- 0.4 d32-1--incoming 3 0 0 --p--- 0.0 /var/log/xen/xl-d32-1.log is just "Waiting for domain d32-1 (domid 9) to die [pid 4045]" (nb: this was a newer attempt than the ones above, to be sure I was looking at the right log, so the domid''s don''t match, 9 =d32-1 not the incoming one). There is no xl log for the incoming domain. Also it''d be worth pinging/CCing Shriram next time to get him to sanity test the Remus cases too. I''m in the middle of reviewing #5/19 (the meat), I''ll keep going although I doubt I''ll spot the cause of this... Ian.
Ian Campbell
2012-Jun-13 10:22 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Wed, 2012-06-13 at 09:59 +0100, Ian Campbell wrote:> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote: > > This is v3 of my series to asyncify save/restore, rebased to current > > tip, retested, and with all comments addressed. > > There''s quite a lot of combinations which need testing here (PV, HVM, > HVM w/ stub dm, old vs new qemu etc etc), which of those have you tried? > > I tried a simple localhost migrate of a PV guest and: > # xl -vvv migrate d32-1 localhost > migration target: Ready to receive domain. > Saving to migration stream new xl format (info 0x0/0x0/3541) > libxl: debug: libxl.c:722:libxl_domain_suspend: ao 0x8069720: create: how=(nil) callback=(nil) poller=0x80696c8 > Loading new save file <incoming migration stream> (new xl fmt info 0x0/0x0/3541) > Savefile contains xl domain config > libxl: debug: libxl_dom.c:969:libxl__toolstack_save: domain=2 toolstack data size=8 > libxl: debug: libxl.c:745:libxl_domain_suspend: ao 0x8069720: inprogress: poller=0x80696c8, flags=i > libxl-save-helper: debug: starting save: Success > xc: detail: Had 0 unexplained entries in p2m table > xc: Saving memory: iter 0 (last sent 0 skipped 0): 0/131072 0% > > at which point it appears to just stop. > > # strace -p 2872 # /usr/lib/xen/bin/libxl-save-helper --save-domain 8 2 0 0 1 0 0 12 8 72 > Process 2872 attached - interrupt to quit > write(8, 0xb5d31000, 1974272^C <unfinished ...> > Process 2872 detached > # strace -p 2866 # /usr/lib/xen/bin/libxl-save-helper --restore-domain 0 3 1 0 2 0 0 1 0 0 0The first zero here is restore_fd, I think. But I read in the comment in the helper: > + * The helper talks on stdin and stdout, in binary in machine > + * endianness. The helper speaks first, and only when it has a > + * callback to make. It writes a 16-bit number being the message > + * length, and then the message body. So restore_fd == stdin => running two protocols over the same fd?> Process 2866 attached - interrupt to quit > read(0, ^C <unfinished ...> > # strace -p 4070 # xl -vvv migrate d32-1 localhost > Process 4070 attached - interrupt to quit > restart_syscall(<... resuming interrupted call ...> > # strace -p 4074 # xl migrate-receive > Process 4074 attached - interrupt to quit > restart_syscall(<... resuming interrupted call ...> > > So the saver seems to be blocked writing to fd 8, which is argv[1] == io_fd. > > Also FWIW: > # xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 511 4 r----- 24.5 > d32-1 2 128 4 -b---- 0.4 > d32-1--incoming 3 0 0 --p--- 0.0 > > /var/log/xen/xl-d32-1.log is just "Waiting for domain d32-1 (domid 9) to > die [pid 4045]" (nb: this was a newer attempt than the ones above, to be > sure I was looking at the right log, so the domid''s don''t match, 9 => d32-1 not the incoming one). There is no xl log for the incoming domain. > > Also it''d be worth pinging/CCing Shriram next time to get him to sanity > test the Remus cases too. > > I''m in the middle of reviewing #5/19 (the meat), I''ll keep going > although I doubt I''ll spot the cause of this... > > Ian. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Ian Campbell
2012-Jun-13 10:30 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Wed, 2012-06-13 at 11:22 +0100, Ian Campbell wrote:> On Wed, 2012-06-13 at 09:59 +0100, Ian Campbell wrote: > > On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote: > > > This is v3 of my series to asyncify save/restore, rebased to current > > > tip, retested, and with all comments addressed. > > > > There''s quite a lot of combinations which need testing here (PV, HVM, > > HVM w/ stub dm, old vs new qemu etc etc), which of those have you tried? > > > > I tried a simple localhost migrate of a PV guest and: > > # xl -vvv migrate d32-1 localhost > > migration target: Ready to receive domain. > > Saving to migration stream new xl format (info 0x0/0x0/3541) > > libxl: debug: libxl.c:722:libxl_domain_suspend: ao 0x8069720: create: how=(nil) callback=(nil) poller=0x80696c8 > > Loading new save file <incoming migration stream> (new xl fmt info 0x0/0x0/3541) > > Savefile contains xl domain config > > libxl: debug: libxl_dom.c:969:libxl__toolstack_save: domain=2 toolstack data size=8 > > libxl: debug: libxl.c:745:libxl_domain_suspend: ao 0x8069720: inprogress: poller=0x80696c8, flags=i > > libxl-save-helper: debug: starting save: Success > > xc: detail: Had 0 unexplained entries in p2m table > > xc: Saving memory: iter 0 (last sent 0 skipped 0): 0/131072 0% > > > > at which point it appears to just stop. > > > > # strace -p 2872 # /usr/lib/xen/bin/libxl-save-helper --save-domain 8 2 0 0 1 0 0 12 8 72 > > Process 2872 attached - interrupt to quit > > write(8, 0xb5d31000, 1974272^C <unfinished ...> > > Process 2872 detached > > # strace -p 2866 # /usr/lib/xen/bin/libxl-save-helper --restore-domain 0 3 1 0 2 0 0 1 0 0 0 > > The first zero here is restore_fd, I think. But I read in the comment in > the helper: > > + * The helper talks on stdin and stdout, in binary in machine > > + * endianness. The helper speaks first, and only when it has a > > + * callback to make. It writes a 16-bit number being the message > > + * length, and then the message body. > > So restore_fd == stdin => running two protocols over the same fd?Oh, right, migrate-receive takes the migration fd on stdin doesn''t it, so that''s where it comes from. I still suspect it is wrong. Might need to dup the input onto a safe fd? BTW, since I''ve been ctrl-c''ing "xl migrate" a bunch I noticed that we seem to leak an "xl migrate-receive" and the restore side helper process. Probably pre-existing but I thought it worth mentioning.> > > Process 2866 attached - interrupt to quit > > read(0, ^C <unfinished ...> > > # strace -p 4070 # xl -vvv migrate d32-1 localhost > > Process 4070 attached - interrupt to quit > > restart_syscall(<... resuming interrupted call ...> > > # strace -p 4074 # xl migrate-receive > > Process 4074 attached - interrupt to quit > > restart_syscall(<... resuming interrupted call ...> > > > > So the saver seems to be blocked writing to fd 8, which is argv[1] == io_fd. > > > > Also FWIW: > > # xl list > > Name ID Mem VCPUs State Time(s) > > Domain-0 0 511 4 r----- 24.5 > > d32-1 2 128 4 -b---- 0.4 > > d32-1--incoming 3 0 0 --p--- 0.0 > > > > /var/log/xen/xl-d32-1.log is just "Waiting for domain d32-1 (domid 9) to > > die [pid 4045]" (nb: this was a newer attempt than the ones above, to be > > sure I was looking at the right log, so the domid''s don''t match, 9 => > d32-1 not the incoming one). There is no xl log for the incoming domain. > > > > Also it''d be worth pinging/CCing Shriram next time to get him to sanity > > test the Remus cases too. > > > > I''m in the middle of reviewing #5/19 (the meat), I''ll keep going > > although I doubt I''ll spot the cause of this... > > > > Ian. > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Ian Campbell
2012-Jun-13 10:38 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
On Wed, 2012-06-13 at 09:59 +0100, Ian Campbell wrote:> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote: > > This is v3 of my series to asyncify save/restore, rebased to current > > tip, retested, and with all comments addressed. > > There''s quite a lot of combinations which need testing here (PV, HVM, > HVM w/ stub dm, old vs new qemu etc etc), which of those have you tried? > > I tried a simple localhost migrate of a PV guest and:As a separate issue in "xl -vvv save d32-1 /scratch/SAVE" I no longer see the progress messages (xxx% complete etc). I saw something which looked like support for logging progress messages so I was a bit surprised... The complete output is # xl -vvv save d32-1 /scratch/SAVE Saving to /scratch/SAVE new xl format (info 0x0/0x0/3541) libxl: debug: libxl.c:722:libxl_domain_suspend: ao 0x80696c8: create: how=(nil) callback=(nil) poller=0x8069708 libxl: debug: libxl_dom.c:969:libxl__toolstack_save: domain=14 toolstack data size=8 libxl: debug: libxl.c:745:libxl_domain_suspend: ao 0x80696c8: inprogress: poller=0x8069708, flags=i libxl-save-helper: debug: starting save: Success libxl: debug: libxl_dom.c:798:libxl__domain_suspend_common_callback: issuing PV suspend request via XenBus control node libxl: debug: libxl_dom.c:802:libxl__domain_suspend_common_callback: wait for the guest to acknowledge suspend request libxl: debug: libxl_dom.c:849:libxl__domain_suspend_common_callback: guest acknowledged suspend request libxl: debug: libxl_dom.c:853:libxl__domain_suspend_common_callback: wait for the guest to suspend libxl: debug: libxl_dom.c:867:libxl__domain_suspend_common_callback: guest has suspended xc: detail: Had 0 unexplained entries in p2m table xc: debug: outbuf_write: 4194304 > 4177904@12599312 xc: debug: outbuf_write: 2527232 > 2514932@14262284 xc: debug: outbuf_write: 4194304 > 1650672@15126544 xc: debug: outbuf_write: 4194304 > 4182004@12595212 xc: debug: outbuf_write: 4194304 > 4182004@12595212 xc: debug: outbuf_write: 4194304 > 4182004@12595212 xc: debug: outbuf_write: 4194304 > 4182004@12595212 xc: debug: outbuf_write: 3424256 > 3411956@13365260 xc: debug: outbuf_write: 753664 > 753648@16023568 xc: detail: delta 90669ms, dom0 2%, target 0%, sent 11Mb/s, dirtied 0Mb/s 0 pages xc: detail: Total pages sent= 32768 (0.25x) xc: detail: (of which 0 were fixups) xc: detail: All memory is saved xc: detail: Save exit rc=0 libxl-save-helper: debug: complete r=0: Success libxl: debug: libxl_event.c:1434:libxl__ao_complete: ao 0x80696c8: complete, rc=0 libxl: debug: libxl_event.c:1406:libxl__ao__destroy: ao 0x80696c8: destroy libxl: debug: libxl_dm.c:1139:libxl__destroy_device_model: Device Model signaled Restore worked BTW but was a bit spammy with: xc: debug: batch 1024 (maybe not your fault...) Ian.
Ian Campbell
2012-Jun-13 11:04 UTC
Re: [PATCH 05/19] libxl: domain save/restore: run in a separate process
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> libxenctrl expects to be able to simply run the save or restore > operation synchronously. This won''t work well in a process which is > trying to handle multiple domains. > > The options are: > > - Block such a whole process (eg, the whole of libvirt) while > migration completes (or until it fails). > > - Create a thread to run xc_domain_save and xc_domain_restore on. > This is quite unpalatable. Multithreaded programming is error > prone enough without generating threads in libraries, particularly > if the thread does some very complex operation. > > - Fork and run the operation in the child without execing. This is > no good because we would need to negotiate with the caller about > fds we would inherit (and we might be a very large process). > > - Fork and exec a helper. > > Of these options the latter is the most palatable. > > Consequently: > > * A new helper program libxl-save-helper (which does both save and > restore). It will be installed in /usr/lib/xen/bin. It does not > link against libxl, only libxc, and its error handling does not > need to be very advanced. It does contain a plumbing through of > the logging interface into the callback stream. > > * A small ad-hoc protocol between the helper and libxl which allows > log messages and the libxc callbacks to be passed up and down. > Protocol doc comment is in libxl_save_helper.c. > > * To avoid a lot of tedium the marshalling boilerplate (stubs for the > helper and the callback decoder for libxl) is generated with a > small perl script. > > * Implement new functionality to spawn the helper, monitor its > output, provide responses, and check on its exit status. > > * The functions libxl__xc_domain_restore_done and > libxl__xc_domain_save_done now turn out to want be called in the > same place. So make their state argument a void* so that the two > functions are type compatible. > > The domain save path still writes the qemu savefile synchronously. > This will need to be fixed in a subsequent patch. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> > > Changes in v3: > * Suppress errno value in debug message when helper reports successful > completion. > * Significant consequential changes to cope with changes to > earlier patches in the series. > > Changes in v2: > * Helper path can be overridden by an environment variable for testing. > * Add a couple of debug logging messages re toolstack data. > * Fixes from testing. > * Helper protocol message lengths (and numbers) are 16-bit which > more clearly avoids piling lots of junk on the stack. > * Merged with remus changes. > * Callback implementations in libxl now called via pointers > so remus can have its own callbacks. > * Better namespace prefixes on autogenerated names etc. > * Autogenerator can generate debugging printfs too. > ---[...]> diff --git a/.hgignore b/.hgignore > index 27d8f79..05304ea 100644 > --- a/.hgignore > +++ b/.hgignore > @@ -180,9 +180,11 @@ > ^tools/libxl/_.*\.c$ > ^tools/libxl/libxlu_cfg_y\.output$ > ^tools/libxl/xl$ > +^tools/libxl/libxl-save-helper$ > ^tools/libxl/testidl$ > ^tools/libxl/testidl\.c$ > ^tools/libxl/tmp\..*$ > +^tools/libxl/.*\.new$Our naming scheme for these sorts of temporary files is rather in consistent (including an existing use of .new), oh well... [...]> +SAVE_HELPER_OBJS = libxl_save_helper.o _libxl_save_msgs_helper.o > +$(SAVE_HELPER_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenlight) > + > testidl.o: CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenlight) > testidl.c: libxl_types.idl gentest.py libxl.h $(AUTOINCS) > $(PYTHON) gentest.py libxl_types.idl testidl.c.new > @@ -117,6 +122,12 @@ _libxl_list.h: $(XEN_INCLUDE)/xen-external/bsd-sys-queue-h-seddery $(XEN_INCLUDE > perl $^ --prefix=libxl >$@.new > $(call move-if-changed,$@.new,$@) > > +_libxl_save_msgs_helper.c _libxl_save_msgs_callout.c \ > +_libxl_save_msgs_helper.h _libxl_save_msgs_callout.h: \ > + libxl_save_msgs_gen.pl > + $(PERL) -w $< $@ >$@.new > + $(call move-if-changed,$@.new,$@) > + > libxl.h: _libxl_types.h > libxl_json.h: _libxl_types_json.h > libxl_internal.h: _libxl_types_internal.h _paths.h > @@ -159,6 +170,9 @@ libxlutil.a: $(LIBXLU_OBJS) > xl: $(XL_OBJS) libxlutil.so libxenlight.so > $(CC) $(LDFLAGS) -o $@ $(XL_OBJS) libxlutil.so $(LDLIBS_libxenlight) $(LDLIBS_libxenctrl) -lyajl $(APPEND_LDFLAGS) > > +libxl-save-helper: $(SAVE_HELPER_OBJS) libxenlight.so > + $(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)The changelog says that libxl-save-helper doesn''t link libxenlight but it is declared as a dependency here and CFLAGS_libxenlight is included in SAVE_HELPER_OBJS'' CFLAGS above.> + > testidl: testidl.o libxlutil.so libxenlight.so > $(CC) $(LDFLAGS) -o $@ testidl.o libxlutil.so $(LDLIBS_libxenlight) $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)[...]> @@ -170,6 +184,7 @@ install: all > $(INSTALL_DIR) $(DESTDIR)$(BASH_COMPLETION_DIR) > $(INSTALL_DIR) $(DESTDIR)$(XEN_RUN_DIR) > $(INSTALL_PROG) xl $(DESTDIR)$(SBINDIR) > + $(INSTALL_PROG) libxl-save-helper $(DESTDIR)$(PRIVATE_BINDIR)This needs an INSTALL_DIR for $(DESTDIR)$(PRIVATE_BINDIR) to support "make -C tools/libxl DESTDIR=xxx install" else: install: cannot create regular file `/tmp/tmplKa0Y7/usr/lib/xen/bin'': No such file or directory> $(INSTALL_PROG) libxenlight.so.$(MAJOR).$(MINOR) $(DESTDIR)$(LIBDIR) > ln -sf libxenlight.so.$(MAJOR).$(MINOR) $(DESTDIR)$(LIBDIR)/libxenlight.so.$(MAJOR) > ln -sf libxenlight.so.$(MAJOR) $(DESTDIR)$(LIBDIR)/libxenlight.so > > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index b48452c..fea0c94 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -465,16 +465,20 @@ static inline char *restore_helper(libxl__gc *gc, uint32_t domid, > } > > int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > - uint32_t size, void *data) > + uint32_t size, void *user) > { > - libxl__gc *gc = data; > - libxl_ctx *ctx = gc->owner; > + libxl__save_helper_state *shs = user; > + libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs); > + STATE_AO_GC(dcs->ao); > + libxl_ctx *ctx = CTX;Any reason you didn''t s/ctx/CTX/ in one of your preparatory patches?> int i, ret; > const uint8_t *ptr = buf; > uint32_t count = 0, version = 0; > struct libxl__physmap_info* pi; > char *xs_path;[...]> diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c > index 1b481ab..a39f434 100644 > --- a/tools/libxl/libxl_save_callout.c > +++ b/tools/libxl/libxl_save_callout.c > @@ -16,6 +16,19 @@ > > #include "libxl_internal.h" > > +static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs, > + const char *mode_arg, int data_fd, > + const unsigned long *argnums, int num_argnums); > + > +static void helper_failed(libxl__egc*, libxl__save_helper_state *shs, int rc); > +static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev, > + int fd, short events, short revents); > +static void helper_exited(libxl__egc *egc, libxl__ev_child *ch, > + pid_t pid, int status); > +static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs); > + > +/*----- entrypoints -----*/ > + > void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs, > int hvm, int pae, int superpages, > int no_incr_generationid) > @@ -27,22 +40,319 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs, > const int restore_fd = dcs->restore_fd; > libxl__domain_build_state *const state = &dcs->build_state; > > - int r = xc_domain_restore(CTX->xch, restore_fd, domid, > - state->store_port, &state->store_mfn, > - state->store_domid, state->console_port, > - &state->console_mfn, state->console_domid, > - hvm, pae, superpages, no_incr_generationid, > - &state->vm_generationid_addr, &dcs->callbacks); > - libxl__xc_domain_restore_done(egc, dcs, 0, r, errno); > + unsigned cbflags = libxl__srm_callout_enumcallbacks_restore > + (&dcs->shs.callbacks.restore.a); > + > + const unsigned long argnums[] = { > + restore_fd, domid, > + state->store_port, > + state->store_domid, state->console_port, > + state->console_domid, > + hvm, pae, superpages, no_incr_generationid, > + cbflags, > + }; > + > + dcs->shs.ao = ao; > + dcs->shs.domid = domid; > + dcs->shs.recv_callback = libxl__srm_callout_received_restore; > + dcs->shs.completion_callback = libxl__xc_domain_restore_done; > + dcs->shs.caller_state = dcs; > + dcs->shs.need_results = 1; > + dcs->shs.toolstack_data_file = 0; > + > + run_helper(egc, &dcs->shs, "--restore-domain", restore_fd, > + argnums, ARRAY_SIZE(argnums)); > } > > void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss, > unsigned long vm_generationid_addr) > { > STATE_AO_GC(dss->ao); > - int r; > + int r, rc; > + uint32_t toolstack_data_len; > + > + /* Resources we need to free */ > + uint8_t *toolstack_data_buf = 0; > + > + unsigned cbflags = libxl__srm_callout_enumcallbacks_save > + (&dss->shs.callbacks.save.a); > + > + r = libxl__toolstack_save(dss->domid, &toolstack_data_buf, > + &toolstack_data_len, dss);This seems to be called twice? I''m looking at the source after the full series is applied and I see dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save; in libxl__domain_suspend as well as this call here. The callback one seems to be otherwise unused.> +static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs, > + const char *mode_arg, int data_fd, > + const unsigned long *argnums, int num_argnums) > +{ > + STATE_AO_GC(shs->ao); > + const char *args[3 + num_argnums]; > + const char **arg = args; > + int i, rc; > + > + /* Resources we must free */ > + libxl__carefd *childs_pipes[2] = { 0,0 }; > + > + /* Convenience aliases */ > + const uint32_t domid = shs->domid; > + > + shs->rc = 0; > + shs->completed = 0; > + shs->pipes[0] = shs->pipes[1] = 0; > + libxl__ev_fd_init(&shs->readable); > + libxl__ev_child_init(&shs->child); > + > + shs->stdin_what = GCSPRINTF("domain %"PRIu32" save/restore helper" > + " stdin pipe", domid); > + shs->stdout_what = GCSPRINTF("domain %"PRIu32" save/restore helper" > + " stdout pipe", domid); > + > + *arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC "/" "libxl-save-helper"; > + *arg++ = mode_arg; > + for (i=0; i<num_argnums; i++) > + *arg++ = GCSPRINTF("%lu", argnums[i]); > + *arg++ = 0; > + assert(arg == args + ARRAY_SIZE(args)); > + > + libxl__carefd_begin(); > + for (i=0; i<2; i++) { > + int fds[2]; > + if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; } > + childs_pipes[i] = libxl__carefd_record(CTX, fds[i]); > + shs->pipes[i] = libxl__carefd_record(CTX, fds[!i]);Using !i here is clever, but I had to deploy a pen to be sure the assignments were all correct. Open coding this simple loop would also give you the opportunity the have comments like "/* This is the helper processes''s stdout */" etc in the appropriate place to aid in comprehension when checking the correct end of each pipe goes in the right place.> + } > + libxl__carefd_unlock(); > + > + pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited); > + if (!pid) { > + libxl_fd_set_cloexec(CTX, data_fd, 0); > + libxl__exec(gc, > + libxl__carefd_fd(childs_pipes[0]), > + libxl__carefd_fd(childs_pipes[1]), > + -1, > + args[0], (char**)args, 0); > + } > + > + libxl__carefd_close(childs_pipes[0]); > + libxl__carefd_close(childs_pipes[1]); > + > + rc = libxl__ev_fd_register(gc, &shs->readable, helper_stdout_readable, > + libxl__carefd_fd(shs->pipes[1]), POLLIN|POLLPRI); > + if (rc) goto out; > + return; > + > + out: > + libxl__carefd_close(childs_pipes[0]); > + libxl__carefd_close(childs_pipes[1]); > + helper_failed(egc, shs, rc);; > +} > + > +static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs, > + int rc) > +{ > + STATE_AO_GC(shs->ao); > + > + if (!shs->rc) > + shs->rc = rc; > + > + libxl__ev_fd_deregister(gc, &shs->readable); > + > + if (!libxl__ev_child_inuse(&shs->child)) { > + helper_done(egc, shs); > + return; > + } > + > + int r = kill(shs->child.pid, SIGKILL); > + if (r) LOGE(WARN, "failed to kill save/restore helper [%lu]", > + (unsigned long)shs->child.pid); > +} > + > +static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev, > + int fd, short events, short revents) > +{ > + libxl__save_helper_state *shs = CONTAINER_OF(ev, *shs, readable); > + STATE_AO_GC(shs->ao); > + int rc, errnoval; > + > + if (revents & POLLPRI) { > + LOG(ERROR, "%s signaled POLLERR", shs->stdout_what);signalled ? (it''s not impossible that my spell checker is wrong, google seem to think both are ok) The if says POLLPRI but the log says POLLERR?> + rc = ERROR_FAIL; > + out: > + /* this is here because otherwise we bypass the decl of msg[] */I don''t get this comment, why can''t "out:" be in the normal place after the non-error return?> + helper_failed(egc, shs, rc); > + return; > + } > + > + uint16_t msglen; > + errnoval = libxl_read_exactly(CTX, fd, &msglen, sizeof(msglen), > + shs->stdout_what, "ipc msg header"); > + if (errnoval) { rc = ERROR_FAIL; goto out; } > + > + unsigned char msg[msglen]; > + errnoval = libxl_read_exactly(CTX, fd, msg, msglen, > + shs->stdout_what, "ipc msg body"); > + if (errnoval) { rc = ERROR_FAIL; goto out; } > + > + shs->egc = egc; > + shs->recv_callback(msg, msglen, shs); > + shs->egc = 0; > + return; > +} > + > +static void helper_exited(libxl__egc *egc, libxl__ev_child *ch, > + pid_t pid, int status) > +{ > + libxl__save_helper_state *shs = CONTAINER_OF(ch, *shs, child); > + STATE_AO_GC(shs->ao); > + > + /* Convenience aliases */ > + const uint32_t domid = shs->domid; > + > + const char *what > + GCSPRINTF("domain %"PRIu32" save/restore helper", domid); > + > + if (status) { > + libxl_report_child_exitstatus(CTX, XTL_ERROR, what, pid, status); > + shs->rc = ERROR_FAIL; > + } > + > + if (shs->need_results) { > + if (!shs->rc) > + LOG(ERROR,"%s exited without providing results",what); > + shs->rc = ERROR_FAIL; > + } > + > + if (!shs->completed) { > + if (!shs->rc) > + LOG(ERROR,"%s exited without signaling completion",what);signalling again, same caveat> + shs->rc = ERROR_FAIL; > + } > + > + helper_done(egc, shs); > + return; > +}[...]> +/*----- generic helpers for the autogenerated code -----*/[...]> diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c > new file mode 100644 > index 0000000..d29f1f7 > --- /dev/null > +++ b/tools/libxl/libxl_save_helper.c > @@ -0,0 +1,279 @@ > +/* > + * Copyright (C) 2012 Citrix Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU Lesser General Public License as published > + * by the Free Software Foundation; version 2.1 only. with the special > + * exception on linking described in file LICENSE.LGPL, or perhaps GPL since this is a helper? (I suppose the same argument could be made for xl itself) [...]> +/*----- helper functions called by autogenerated stubs -----*/ > + > +unsigned char * helper_allocbuf(int len, void *user) > +{ > + return xmalloc(len); > +} > + > +static void transmit(const unsigned char *msg, int len, void *user) > +{ > + while (len) { > + int r = write(1, msg, len); > + if (r<0) { perror("write"); exit(-1); } > + assert(r >= 0); > + assert(r <= len); > + len -= r; > + msg += r; > + } > +} > + > +void helper_transmitmsg(unsigned char *msg_freed, int len_in, void *user) > +{ > + assert(len_in < 64*1024); > + uint16_t len = len_in; > + transmit((const void*)&len, sizeof(len), user); > + transmit(msg_freed, len, user); > + free(msg_freed); > +} > + > +int helper_getreply(void *user) > +{ > + int v; > + int r = read_exactly(0, &v, sizeof(v)); > + if (r<=0) exit(-2);You use -1 a lot but here you use -2, are we supposed to be able to infer something from the specific error code?> + return v; > +} > + > +/*----- other callbacks -----*/ > + > +static int toolstack_save_fd; > +static uint32_t toolstack_save_len;[...]> diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl > new file mode 100755 > index 0000000..cd0837e > --- /dev/null > +++ b/tools/libxl/libxl_save_msgs_gen.pl > @@ -0,0 +1,393 @@ > +#!/usr/bin/perl -w > + > +use warnings; > +use strict; > +use POSIX; > + > +our $debug = 0; # produce copius debugging output at run-time?copious (which is probably the most useful feedback you are going to get from me on a Perl script ;-))> + > +our @msgs = ( > + # flags: > + # s - applicable to save > + # r - applicable to restore > + # c - function pointer in callbacks struct rather than fixed function > + # x - function pointer is in struct {save,restore}_callbacks > + # and its null-ness needs to be passed through to the helper''s xc > + # W - needs a return value; callback is synchronous > + [ 1, ''sr'', "log", [qw(uint32_t level > + uint32_t errnoval > + STRING context > + STRING formatted)] ], > + [ 2, ''sr'', "progress", [qw(STRING context > + STRING doing_what), > + ''unsigned long'', ''done'', > + ''unsigned long'', ''total''] ], > + [ 3, ''scxW'', "suspend", [] ], > + [ 4, ''scxW'', "postcopy", [] ], > + [ 5, ''scxW'', "checkpoint", [] ], > + [ 6, ''scxW'', "switch_qemu_logdirty", [qw(int domid > + unsigned enable)] ], > + # toolstack_save done entirely `by hand'' > + [ 7, ''rcxW'', "toolstack_restore", [qw(uint32_t domid > + BLOCK tsdata)] ], > + [ 8, ''r'', "restore_results", [''unsigned long'', ''store_mfn'', > + ''unsigned long'', ''console_mfn'', > + ''unsigned long'', ''genidad''] ], > + [ 9, ''srW'', "complete", [qw(int retval > + int errnoval)] ], > +); > + > +#---------------------------------------- > + > +our %cbs; > +our %func; > +our %func_ah; > +our @outfuncs; > +our %out_decls; > +our %out_body; > +our %msgnum_used; > + > +die unless @ARGV==1; > +die if $ARGV[0] =~ m/^-/; > + > +our ($intendedout) = @ARGV; > + > +$intendedout =~ m/([a-z]+)\.([ch])$/ or die; > +my ($want_ah, $ch) = ($1, $2); > + > +my $declprefix = ''''; > + > +foreach my $ah (qw(callout helper)) { > + $out_body{$ah} .= <<END.($ah eq ''callout'' ? <<END : <<END);[...]> +END[...]> +END[...]> +ENDI think I can infer what this does, but wow ;-)> +}[...]
Ian Campbell
2012-Jun-13 11:15 UTC
Re: [PATCH 07/19] libxl: provide libxl__xs_*_checked and libxl__xs_transaction_*
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> These useful utility functions make dealing with xenstore a little > less painful. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com> (minor observation and a typo below) [...]> +/* On success, *result_out came from the gc. > + * On error, *result_out is undefined. > + * ENOENT counts as success but sets *result_out=0 > + */ > +int libxl__xs_read_checked(libxl__gc *gc, xs_transaction_t t, > + const char *path, const char **result_out); > + > +/* Does not include a trailing null. > + * May usefully be combined with GCSPRINTF if the format string > + * behaviour of libxl__xs_write is desirable. */ > +int libxl__xs_write_checked(libxl__gc *gc, xs_transaction_t t, > + const char *path, const char *string);I suppose in the future we might consider merging this with libxl__xs_write -- there''s no reason not to always log a failed write, is there?> + > +/* ENOENT is not an error (even if the parent directories don''t exist) */ > +int libxl__xs_rm_checked(libxl__gc *gc, xs_transaction_t t, const char *path); > + > +/* Transaction functions, best used together. > + * The caller should initialise *t to 0 (XBT_NULL) before calling start. > + * Each function leaves *t!=0 iff the transaction needs cleaning up. > + * > + * libxl__xs_transaction_commit returns: > + * <0 failure - a libxl error code > + * +1 commit conflict; transaction has been destroyed and caller > + * must go round again (call _start again and retry) > + * 0 commited successfullycommitted> + */ > +int libxl__xs_transaction_start(libxl__gc *gc, xs_transaction_t *t); > +int libxl__xs_transaction_commit(libxl__gc *gc, xs_transaction_t *t); > +void libxl__xs_transaction_abort(libxl__gc *gc, xs_transaction_t *t); > + > + > +Ian.
Ian Jackson
2012-Jun-13 11:27 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process
Ian Campbell writes ("Re: [Xen-devel] [PATCH v3 00/18] libxl: domain
save/restore: run in a separate process"):> On Wed, 2012-06-13 at 09:59 +0100, Ian Campbell wrote:
> > There''s quite a lot of combinations which need testing here
(PV, HVM,
> > HVM w/ stub dm, old vs new qemu etc etc), which of those have you
tried?
I haven''t tried stub dm or new qemu. I have tried save/restore of
both PV and HVM. I have also done localhost migration - although
perhaps not as often or in as many combinations.
> > I tried a simple localhost migrate of a PV guest and:
>
> As a separate issue in "xl -vvv save d32-1 /scratch/SAVE" I no
longer
> see the progress messages (xxx% complete etc). I saw something which
> looked like support for logging progress messages so I was a bit
> surprised...
I''ll look into this. I think you''re right about the fd bug
too.
I don''t know why this didn''t go wrong for me.
Ian.
Ian Campbell
2012-Jun-13 12:52 UTC
Re: [PATCH 08/19] libxl: wait for qemu to acknowledge logdirty command
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> The current migration code in libxl instructs qemu to start or stop > logdirty, but it does not wait for an acknowledgement from qemu before > continuing. This might lead to memory corruption (!) > > Fix this by waiting for qemu to acknowledge the command. > > Unfortunately the necessary ao arrangements for waiting for this > command are unique because qemu has a special protocol for this > particular operation. > > Also, this change means that the switch_qemu_logdirty callback > implementation in libxl can no longer synchronously produce its return > value, as it now needs to wait for xenstore. So we tell the > marshalling code generator that it is a message which does not need a > reply. This turns the callback function called by the marshaller into > one which returns void; the callback function arranges to later > explicitly sends the reply to the helper, when the xs watch triggers > and the appropriate value is read from xenstore. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> > --- > tools/libxl/libxl_dom.c | 176 +++++++++++++++++++++++++++++++++--- > tools/libxl/libxl_internal.h | 18 ++++- > tools/libxl/libxl_save_callout.c | 8 ++ > tools/libxl/libxl_save_msgs_gen.pl | 7 +- > 4 files changed, 193 insertions(+), 16 deletions(-) > > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index a597627..d5ac79f 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -528,30 +528,180 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > static void domain_suspend_done(libxl__egc *egc, > libxl__domain_suspend_state *dss, int rc); > > -/*----- callbacks, called by xc_domain_save -----*/ > +/*----- complicated callback, called by xc_domain_save -----*/ > + > +/* > + * We implement the other end of protocol for controlling qemu-dm''s > + * logdirty. There is no documentation for this protocol, but our > + * counterparty''s implementation is in > + * qemu-xen-traditional.git:xenstore.c in the function > + * xenstore_process_logdirty_event > + */ > + > +static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev, > + const struct timeval *requested_abs); > +static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*, > + const char *watch_path, const char *event_path); > +static void switch_logdirty_done(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int ok); > > -int libxl__domain_suspend_common_switch_qemu_logdirty > +static void logdirty_init(libxl__logdirty_switch *lds) > +{ > + lds->cmd_path = 0; > + libxl__ev_xswatch_init(&lds->watch); > + libxl__ev_time_init(&lds->timeout);I meant to mention this once before when reviewing some patch or other but I''m not sure I actually did, so at the risk of repeating myself: This watch with timeout pattern seems to be reasonably common (in fact, I''m not sure we have any watches without a timeout). It might be a candidate for a specific helper routine in the future? [...]> +static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev, > + const struct timeval *requested_abs) > +{ > + libxl__domain_suspend_state *dss = CONTAINER_OF(ev, *dss, logdirty.timeout); > + STATE_AO_GC(dss->ao); > + LOG(ERROR,"logdirty switch: wait for device model timed out"); > + switch_logdirty_done(egc,dss,0); > } > > +static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch, > + const char *watch_path, const char *event_path) > +{ > + libxl__domain_suspend_state *dss > + CONTAINER_OF(watch, *dss, logdirty.watch); > + libxl__logdirty_switch *lds = &dss->logdirty; > + STATE_AO_GC(dss->ao); > + const char *got; > + xs_transaction_t t = 0; > + int rc; > + > + for (;;) { > + rc = libxl__xs_transaction_start(gc, &t); > + if (rc) goto out; > + > + rc = libxl__xs_read_checked(gc, t, lds->ret_path, &got); > + if (rc) goto out; > + > + if (!got) { > + rc = +1; > + goto out; > + } > + > + if (strcmp(got, lds->cmd)) { > + LOG(ERROR,"logdirty switch: sent command `%s'' but got reply `%s''" > + " (xenstore paths `%s'' / `%s'')", lds->cmd, got, > + lds->cmd_path, lds->ret_path); > + rc = ERROR_FAIL; > + goto out; > + } > + > + rc = libxl__xs_rm_checked(gc, t, lds->cmd_path); > + if (rc) goto out; > + > + rc = libxl__xs_rm_checked(gc, t, lds->ret_path); > + if (rc) goto out; > + > + rc = libxl__xs_transaction_commit(gc, &t); > + if (!rc) break; > + if (rc<0) goto out; > + } > + > + out: > + /* rc < 0: error > + * rc == 0: ok, we are done > + * rc == +1: need to keep waiting > + */ > + libxl__xs_transaction_abort(gc, &t); > + > + if (!rc) { > + switch_logdirty_done(egc,dss,1); > + } else if (rc < 0) { > + LOG(ERROR,"logdirty switch: response read failed (rc=%d)",rc);Is it only "response read failed" which can cause us to end up here, looks like the read, rm etc could do it? Anyway, I''ve got no substantive comments so: Acked-by: Ian Campbell <ian.campbell@citrix.com>> + switch_logdirty_done(egc,dss,0); > + } > +}[...]
Ian Campbell
2012-Jun-13 12:53 UTC
Re: [PATCH 09/19] libxl: datacopier: provide "prefix data" facility
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> This will be used to write the qemu data banner to the save/migration > stream. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell
2012-Jun-13 12:56 UTC
Re: [PATCH 10/19] libxl: prepare for asynchronous writing of qemu save file
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> * Combine the various calls to libxl__device_model_savefile into one > at the start of libxl__domain_suspend, storing the result in the > dss. Consequently a few functions take a dss instead of some or all > of their other arguments. > > * Make libxl__domain_save_device_model''s API into an asynchronous > style which takes a callback. The function is, however, still > synchronous; it will be made actually async in the next patch. > > * Consequently make libxl__remus_domain_checkpoint_callback into an > asynchronous callback. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com>> --- > tools/libxl/libxl_dom.c | 54 ++++++++++++++++++++++++++---------- > tools/libxl/libxl_internal.h | 18 ++++++++++-- > tools/libxl/libxl_save_msgs_gen.pl | 2 +- > 3 files changed, 55 insertions(+), 19 deletions(-) > > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index d5ac79f..3a53f97 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -702,11 +702,13 @@ static void switch_logdirty_done(libxl__egc *egc, > > /*----- callbacks, called by xc_domain_save -----*/ > > -int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid) > +int libxl__domain_suspend_device_model(libxl__gc *gc, > + libxl__domain_suspend_state *dss) > { > libxl_ctx *ctx = libxl__gc_owner(gc); > int ret = 0; > - const char *filename = libxl__device_model_savefile(gc, domid); > + uint32_t const domid = dss->domid; > + const char *const filename = dss->dm_savefile; > > switch (libxl__device_model_version_running(gc, domid)) { > case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL: { > @@ -875,7 +877,7 @@ int libxl__domain_suspend_common_callback(void *user) > > guest_suspended: > if (dss->hvm) { > - ret = libxl__domain_suspend_device_model(gc, dss->domid); > + ret = libxl__domain_suspend_device_model(gc, dss); > if (ret) { > LOG(ERROR, "libxl__domain_suspend_device_model failed ret=%d", ret); > return 0; > @@ -990,19 +992,32 @@ static int libxl__remus_domain_resume_callback(void *data) > return 1; > } > > -static int libxl__remus_domain_checkpoint_callback(void *data) > +/*----- remus asynchronous checkpoint callback -----*/ > + > +static void remus_checkpoint_dm_saved(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc); > + > +static void libxl__remus_domain_checkpoint_callback(void *data) > { > libxl__domain_suspend_state *dss = data; > + libxl__egc *egc = dss->shs.egc; > STATE_AO_GC(dss->ao); > > /* This would go into tailbuf. */ > - if (dss->hvm && > - libxl__domain_save_device_model(gc, dss->domid, dss->fd)) > - return 0; > + if (dss->hvm) { > + libxl__domain_save_device_model(egc, dss, remus_checkpoint_dm_saved); > + } else { > + remus_checkpoint_dm_saved(egc, dss, 0); > + } > +} > > +static void remus_checkpoint_dm_saved(libxl__egc *egc, > + libxl__domain_suspend_state *dss, int rc) > +{ > /* TODO: Wait for disk and memory ack, release network buffer */ > + /* TODO: make this asynchronous */ > usleep(dss->interval * 1000); > - return 1; > + libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1); > } > > /*----- main code for suspending, in order of execution -----*/ > @@ -1052,6 +1067,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss) > > dss->suspend_eventchn = -1; > dss->guest_responded = 0; > + dss->dm_savefile = libxl__device_model_savefile(gc, domid); > > if (r_info != NULL) { > dss->interval = r_info->interval; > @@ -1101,7 +1117,6 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void, > > /* Convenience aliases */ > const libxl_domain_type type = dss->type; > - const uint32_t domid = dss->domid; > > if (rc) > goto out; > @@ -1119,11 +1134,11 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void, > } > > if (type == LIBXL_DOMAIN_TYPE_HVM) { > - rc = libxl__domain_suspend_device_model(gc, domid); > + rc = libxl__domain_suspend_device_model(gc, dss); > if (rc) goto out; > > - rc = libxl__domain_save_device_model(gc, domid, dss->fd); > - if (rc) goto out; > + libxl__domain_save_device_model(egc, dss, domain_suspend_done); > + return; > } > > rc = 0; > @@ -1132,14 +1147,22 @@ out: > domain_suspend_done(egc, dss, rc); > } > > -int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) > +void libxl__domain_save_device_model(libxl__egc *egc, > + libxl__domain_suspend_state *dss, > + libxl__save_device_model_cb *callback) > { > + STATE_AO_GC(dss->ao); > int rc, fd2 = -1, c; > char buf[1024]; > - const char *filename = libxl__device_model_savefile(gc, domid); > struct stat st; > uint32_t qemu_state_len; > > + dss->save_dm_callback = callback; > + > + /* Convenience aliases */ > + const char *const filename = dss->dm_savefile; > + const int fd = dss->fd; > + > if (stat(filename, &st) < 0) > { > LOG(ERROR, "Unable to stat qemu save file\n"); > @@ -1181,7 +1204,8 @@ int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) > out: > if (fd2 >= 0) close(fd2); > unlink(filename); > - return rc; > + > + dss->save_dm_callback(egc, dss, rc); > } > > static void domain_suspend_done(libxl__egc *egc, > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index 7e0ab11..c7fe9e9 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -824,10 +824,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t domid, > > _hidden int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > uint32_t size, void *data); > -_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid); > -_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, uint32_t domid); > _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid); > -_hidden int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd); > + > _hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid); > > _hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid); > @@ -1869,6 +1867,8 @@ typedef struct libxl__domain_suspend_state libxl__domain_suspend_state; > > typedef void libxl__domain_suspend_cb(libxl__egc*, > libxl__domain_suspend_state*, int rc); > +typedef void libxl__save_device_model_cb(libxl__egc*, > + libxl__domain_suspend_state*, int rc); > > typedef struct libxl__logdirty_switch { > const char *cmd; > @@ -1895,9 +1895,12 @@ struct libxl__domain_suspend_state { > int hvm; > int xcflags; > int guest_responded; > + const char *dm_savefile; > int interval; /* checkpoint interval (for Remus) */ > libxl__save_helper_state shs; > libxl__logdirty_switch logdirty; > + /* private for libxl__domain_save_device_model */ > + libxl__save_device_model_cb *save_dm_callback; > }; > > > @@ -2053,6 +2056,15 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc, > _hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void, > int rc, int retval, int errnoval); > > +/* Each time the dm needs to be saved, we must call suspend and then save */ > +_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, > + libxl__domain_suspend_state *dss); > +_hidden void libxl__domain_save_device_model(libxl__egc *egc, > + libxl__domain_suspend_state *dss, > + libxl__save_device_model_cb *callback); > + > +_hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid); > + > > /* > * Convenience macros. > diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl > index 8832c46..3ac6f80 100755 > --- a/tools/libxl/libxl_save_msgs_gen.pl > +++ b/tools/libxl/libxl_save_msgs_gen.pl > @@ -25,7 +25,7 @@ our @msgs = ( > ''unsigned long'', ''total''] ], > [ 3, ''scxW'', "suspend", [] ], > [ 4, ''scxW'', "postcopy", [] ], > - [ 5, ''scxW'', "checkpoint", [] ], > + [ 5, ''scxA'', "checkpoint", [] ], > [ 6, ''scxA'', "switch_qemu_logdirty", [qw(int domid > unsigned enable)] ], > # toolstack_save done entirely `by hand''
Ian Campbell
2012-Jun-13 12:59 UTC
Re: [PATCH 11/19] libxl: Make libxl__domain_save_device_model asynchronous
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell
2012-Jun-13 12:59 UTC
Re: [PATCH 12/19] libxl: Add a gc to libxl_get_cpu_topology
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> In the next patch we are going to change the definition of NOGC to > require a local variable libxl__gc *gc. > > libxl_get_cpu_topology doesn''t have one but does use NOGC. > Fix this by: > - introducing an `out'' label > - replacing the only call to `return'' with a suitable assignment > to ret and a `goto out''. > - adding uses of GC_INIT and GC_FREE. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com>> --- > tools/libxl/libxl.c | 6 +++++- > 1 files changed, 5 insertions(+), 1 deletions(-) > > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c > index 7de026b..2a31528 100644 > --- a/tools/libxl/libxl.c > +++ b/tools/libxl/libxl.c > @@ -3202,6 +3202,7 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo *physinfo) > > libxl_cputopology *libxl_get_cpu_topology(libxl_ctx *ctx, int *nr) > { > + GC_INIT(ctx); > xc_topologyinfo_t tinfo; > DECLARE_HYPERCALL_BUFFER(xc_cpu_to_core_t, coremap); > DECLARE_HYPERCALL_BUFFER(xc_cpu_to_socket_t, socketmap); > @@ -3214,7 +3215,8 @@ libxl_cputopology *libxl_get_cpu_topology(libxl_ctx *ctx, int *nr) > if (max_cpus == 0) > { > LIBXL__LOG(ctx, XTL_ERROR, "Unable to determine number of CPUS"); > - return NULL; > + ret = NULL; > + goto out; > } > > coremap = xc_hypercall_buffer_alloc > @@ -3259,6 +3261,8 @@ fail: > > if (ret) > *nr = max_cpus; > + out: > + GC_FREE; > return ret; > } >
Ian Campbell
2012-Jun-13 13:08 UTC
Re: [PATCH 14/19] libxl: Get compiler to warn about gc_opt==NULL
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> Since it used to be legal to pass gc_opt==NULL, and there are various > patches floating about and under developmetn which do so, add adevelopment> compiler annotation which makes the build fail when that is done. > > This turns a runtime crash into a build failure, and should ensure > that we don''t accidentally commit a broken combination of patches. > > This is something of an annoying approach because it adds a macro > invocation to the RHS of every declaration of a function taking a > gc_opt. So it should be reverted after Xen 4.2rc1.This patch will itself break the build until a subsequent patch, won''t it? Shuffle it afterwards?> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Wherever it ends up in the sequence: Acked-by: Ian Campbell <ian.campbell@citrix.com>> --- > tools/libxl/libxl_internal.h | 21 +++++++++++++-------- > 1 files changed, 13 insertions(+), 8 deletions(-) > > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index e69482c..8635764 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -453,28 +453,33 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc) > * psuedo-gc. > */ > /* register ptr in gc for free on exit from outermost libxl callframe. */ > -_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */); > + > +#define NN1 __attribute__((nonnull(1))) > + /* It used to be legal to pass NULL for gc_opt. Get the compiler to > + * warn about this if any slip through. */ > + > +_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */) NN1; > /* if this is the outermost libxl callframe then free all pointers in @gc */ > _hidden void libxl__free_all(libxl__gc *gc); > /* allocate and zero @bytes. (similar to a gc''d malloc(3)+memzero()) */ > -_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes); > +_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes) NN1; > /* allocate and zero memory for an array of @nmemb members of @size each. > * (similar to a gc''d calloc(3)). */ > -_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size); > +_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size) NN1; > /* change the size of the memory block pointed to by @ptr to @new_size bytes. > * unlike other allocation functions here any additional space between the > * oldsize and @new_size is not initialised (similar to a gc''d realloc(3)). */ > -_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size); > +_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size) NN1; > /* print @fmt into an allocated string large enoughto contain the result. > * (similar to gc''d asprintf(3)). */ > -_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3); > +_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3) NN1; > /* duplicate the string @c (similar to a gc''d strdup(3)). */ > -_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c); > +_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c) NN1; > /* duplicate at most @n bytes of string @c (similar to a gc''d strndup(3)). */ > -_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n); > +_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n) NN1; > /* strip the last path component from @s and return as a newly allocated > * string. (similar to a gc''d dirname(3)). */ > -_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s); > +_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s) NN1; > > /* Each of these logs errors and returns a libxl error code. > * They do not mind if path is already removed.
Ian Campbell
2012-Jun-13 13:09 UTC
Re: [PATCH 14/19] libxl: Get compiler to warn about gc_opt==NULL
On Wed, 2012-06-13 at 14:08 +0100, Ian Campbell wrote:> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote: > > Since it used to be legal to pass gc_opt==NULL, and there are various > > patches floating about and under developmetn which do so, add a > > development > > > compiler annotation which makes the build fail when that is done. > > > > This turns a runtime crash into a build failure, and should ensure > > that we don''t accidentally commit a broken combination of patches. > > > > This is something of an annoying approach because it adds a macro > > invocation to the RHS of every declaration of a function taking a > > gc_opt. So it should be reverted after Xen 4.2rc1. > > This patch will itself break the build until a subsequent patch, won''t > it? Shuffle it afterwards?Nevermind, I accidentally skipped #13...> > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> > > Wherever it ends up in the sequence: > Acked-by: Ian Campbell <ian.campbell@citrix.com> > > > --- > > tools/libxl/libxl_internal.h | 21 +++++++++++++-------- > > 1 files changed, 13 insertions(+), 8 deletions(-) > > > > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > > index e69482c..8635764 100644 > > --- a/tools/libxl/libxl_internal.h > > +++ b/tools/libxl/libxl_internal.h > > @@ -453,28 +453,33 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc) > > * psuedo-gc. > > */ > > /* register ptr in gc for free on exit from outermost libxl callframe. */ > > -_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */); > > + > > +#define NN1 __attribute__((nonnull(1))) > > + /* It used to be legal to pass NULL for gc_opt. Get the compiler to > > + * warn about this if any slip through. */ > > + > > +_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */) NN1; > > /* if this is the outermost libxl callframe then free all pointers in @gc */ > > _hidden void libxl__free_all(libxl__gc *gc); > > /* allocate and zero @bytes. (similar to a gc''d malloc(3)+memzero()) */ > > -_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes); > > +_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes) NN1; > > /* allocate and zero memory for an array of @nmemb members of @size each. > > * (similar to a gc''d calloc(3)). */ > > -_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size); > > +_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size) NN1; > > /* change the size of the memory block pointed to by @ptr to @new_size bytes. > > * unlike other allocation functions here any additional space between the > > * oldsize and @new_size is not initialised (similar to a gc''d realloc(3)). */ > > -_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size); > > +_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size) NN1; > > /* print @fmt into an allocated string large enoughto contain the result. > > * (similar to gc''d asprintf(3)). */ > > -_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3); > > +_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3) NN1; > > /* duplicate the string @c (similar to a gc''d strdup(3)). */ > > -_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c); > > +_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c) NN1; > > /* duplicate at most @n bytes of string @c (similar to a gc''d strndup(3)). */ > > -_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n); > > +_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n) NN1; > > /* strip the last path component from @s and return as a newly allocated > > * string. (similar to a gc''d dirname(3)). */ > > -_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s); > > +_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s) NN1; > > > > /* Each of these logs errors and returns a libxl error code. > > * They do not mind if path is already removed. >
Ian Campbell
2012-Jun-13 13:11 UTC
Re: [PATCH 13/19] libxl: Do not pass NULL as gc_opt; introduce NOGC
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> In 25182:6c3345d7e9d9 the practice of passing NULL to gc-using memory > allocation functions was introduced. However, the arrangements there > were not correct as committed, because the error handling and logging > depends on getting a ctx from the gc - so an allocation error would in > fact result in libxl dereferencing NULL. > > Instead, provide a special dummy gc in the ctx, called `nogc_gc''. It > is marked out specially by having alloc_maxsize==-1, which is > otherwise invalid. > > Functions which need to actually look into the gc use the new test > function gc_is_real (whose purpose is mainly clarity of the code) to > check whether the gc is the dummy one, and do nothing if it is. And > we provide a helper macro NOGC which uses the in-scope real gc to find > the ctx and hence the dummy gc (and which replaces the previous > #define NOGC NULL). > > Change all callers which pass 0 or NULL to an allocation function to > use NOGC or &ctx->nogc_gc, as applicable in the context. > > We add a comment near the definition of LIBXL_INIT_GC pointing out > that it isn''t any more the only place a libxl__gc struct is > initialised, for the benefit of anyone changing the contents of gc''s > in the future. > > Also, actually document that libxl__ptr_add is legal with ptr==NULL, > and change a couple of calls not to check for NULL argument. > > Reported-by: Bamvor Jian Zhang <bjzhang@suse.com> > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> > Cc: Bamvor Jian Zhang <bjzhang@suse.com>Acked-by: Ian Campbell <Ian.Campbell@citrix.com>> --- > tools/libxl/libxl.c | 3 +++ > tools/libxl/libxl_aoutils.c | 3 ++- > tools/libxl/libxl_create.c | 2 +- > tools/libxl/libxl_event.c | 5 +++-- > tools/libxl/libxl_exec.c | 2 +- > tools/libxl/libxl_fork.c | 2 +- > tools/libxl/libxl_internal.c | 11 +++++++++-- > tools/libxl/libxl_internal.h | 37 +++++++++++++++++++++++-------------- > tools/libxl/libxl_utils.c | 6 ++---- > tools/libxl/libxl_xshelp.c | 7 ++----- > 10 files changed, 47 insertions(+), 31 deletions(-) > > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c > index 2a31528..a257902 100644 > --- a/tools/libxl/libxl.c > +++ b/tools/libxl/libxl.c > @@ -41,6 +41,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version, > > /* First initialise pointers etc. (cannot fail) */ > > + ctx->nogc_gc.alloc_maxsize = -1; > + ctx->nogc_gc.owner = ctx; > + > LIBXL_TAILQ_INIT(&ctx->occurred); > > ctx->osevent_hooks = 0; > diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c > index 7f8d6d3..99972a2 100644 > --- a/tools/libxl/libxl_aoutils.c > +++ b/tools/libxl/libxl_aoutils.c > @@ -77,6 +77,7 @@ static void datacopier_check_state(libxl__egc *egc, libxl__datacopier_state *dc) > void libxl__datacopier_prefixdata(libxl__egc *egc, libxl__datacopier_state *dc, > const void *data, size_t len) > { > + EGC_GC; > libxl__datacopier_buf *buf; > /* > * It is safe for this to be called immediately after _start, as > @@ -88,7 +89,7 @@ void libxl__datacopier_prefixdata(libxl__egc *egc, libxl__datacopier_state *dc, > > assert(len < dc->maxsz - dc->used); > > - buf = libxl__zalloc(0, sizeof(*buf) - sizeof(buf->buf) + len); > + buf = libxl__zalloc(NOGC, sizeof(*buf) - sizeof(buf->buf) + len); > buf->used = len; > memcpy(buf->buf, data, len); > > diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c > index 00705d8..ab000bc 100644 > --- a/tools/libxl/libxl_create.c > +++ b/tools/libxl/libxl_create.c > @@ -108,7 +108,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, > } > > if (b_info->blkdev_start == NULL) > - b_info->blkdev_start = libxl__strdup(0, "xvda"); > + b_info->blkdev_start = libxl__strdup(NOGC, "xvda"); > > if (b_info->type == LIBXL_DOMAIN_TYPE_HVM) { > if (!b_info->u.hvm.bios) > diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c > index 565d2c2..eb23a93 100644 > --- a/tools/libxl/libxl_event.c > +++ b/tools/libxl/libxl_event.c > @@ -772,7 +772,7 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller, > if (poller->fd_rindices_allocd < maxfd) { > assert(ARRAY_SIZE_OK(poller->fd_rindices, maxfd)); > poller->fd_rindices > - libxl__realloc(0, poller->fd_rindices, > + libxl__realloc(NOGC, poller->fd_rindices, > maxfd * sizeof(*poller->fd_rindices)); > memset(poller->fd_rindices + poller->fd_rindices_allocd, > 0, > @@ -1099,9 +1099,10 @@ void libxl_event_free(libxl_ctx *ctx, libxl_event *event) > libxl_event *libxl__event_new(libxl__egc *egc, > libxl_event_type type, uint32_t domid) > { > + EGC_GC; > libxl_event *ev; > > - ev = libxl__zalloc(0,sizeof(*ev)); > + ev = libxl__zalloc(NOGC,sizeof(*ev)); > ev->type = type; > ev->domid = domid; > > diff --git a/tools/libxl/libxl_exec.c b/tools/libxl/libxl_exec.c > index 082bf44..cfa379c 100644 > --- a/tools/libxl/libxl_exec.c > +++ b/tools/libxl/libxl_exec.c > @@ -280,7 +280,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *ss) > int status, rc; > > libxl__spawn_init(ss); > - ss->ssd = libxl__zalloc(0, sizeof(*ss->ssd)); > + ss->ssd = libxl__zalloc(NOGC, sizeof(*ss->ssd)); > libxl__ev_child_init(&ss->ssd->mid); > > rc = libxl__ev_time_register_rel(gc, &ss->timeout, > diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c > index 9ff99e0..044ddad 100644 > --- a/tools/libxl/libxl_fork.c > +++ b/tools/libxl/libxl_fork.c > @@ -92,7 +92,7 @@ libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd) > libxl__carefd *cf = 0; > > libxl_fd_set_cloexec(ctx, fd, 1); > - cf = libxl__zalloc(NULL, sizeof(*cf)); > + cf = libxl__zalloc(&ctx->nogc_gc, sizeof(*cf)); > cf->fd = fd; > LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry); > return cf; > diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c > index 8139520..fbff7d0 100644 > --- a/tools/libxl/libxl_internal.c > +++ b/tools/libxl/libxl_internal.c > @@ -30,11 +30,16 @@ void libxl__alloc_failed(libxl_ctx *ctx, const char *func, > #undef L > } > > +static int gc_is_real(const libxl__gc *gc) > +{ > + return gc->alloc_maxsize >= 0; > +} > + > void libxl__ptr_add(libxl__gc *gc, void *ptr) > { > int i; > > - if (!gc) > + if (!gc_is_real(gc)) > return; > > if (!ptr) > @@ -66,6 +71,8 @@ void libxl__free_all(libxl__gc *gc) > void *ptr; > int i; > > + assert(gc_is_real(gc)); > + > for (i = 0; i < gc->alloc_maxsize; i++) { > ptr = gc->alloc_ptrs[i]; > gc->alloc_ptrs[i] = NULL; > @@ -104,7 +111,7 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size) > > if (ptr == NULL) { > libxl__ptr_add(gc, new_ptr); > - } else if (new_ptr != ptr && gc != NULL) { > + } else if (new_ptr != ptr && gc_is_real(gc)) { > for (i = 0; i < gc->alloc_maxsize; i++) { > if (gc->alloc_ptrs[i] == ptr) { > gc->alloc_ptrs[i] = new_ptr; > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index f90814a..e69482c 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -277,10 +277,18 @@ struct libxl__poller { > int wakeup_pipe[2]; /* 0 means no fd allocated */ > }; > > +struct libxl__gc { > + /* mini-GC */ > + int alloc_maxsize; /* -1 means this is the dummy non-gc gc */ > + void **alloc_ptrs; > + libxl_ctx *owner; > +}; > + > struct libxl__ctx { > xentoollog_logger *lg; > xc_interface *xch; > struct xs_handle *xsh; > + libxl__gc nogc_gc; > > const libxl_event_hooks *event_hooks; > void *event_hooks_user; > @@ -356,13 +364,6 @@ typedef struct { > > #define PRINTF_ATTRIBUTE(x, y) __attribute__((format(printf, x, y))) > > -struct libxl__gc { > - /* mini-GC */ > - int alloc_maxsize; > - void **alloc_ptrs; > - libxl_ctx *owner; > -}; > - > struct libxl__egc { > /* For event-generating functions only. > * The egc and its gc may be accessed only on the creating thread. */ > @@ -420,6 +421,7 @@ struct libxl__ao { > (gc).alloc_ptrs = 0; \ > (gc).owner = (ctx); \ > } while(0) > + /* NB, also, a gc struct ctx->nogc_gc is initialised in libxl_ctx_alloc */ > > static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc) > { > @@ -438,13 +440,20 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc) > * All pointers returned by these functions are registered for garbage > * collection on exit from the outermost libxl callframe. > * > - * However, where the argument is stated to be "gc_opt", NULL may be > - * passed instead, in which case no garbage collection will occur; the > - * pointer must later be freed with free(). This is for memory > - * allocations of types (b) and (c). > + * However, where the argument is stated to be "gc_opt", &ctx->nogc_gc > + * may be passed instead, in which case no garbage collection will > + * occur; the pointer must later be freed with free(). (Passing NULL > + * for gc_opt is not permitted.) This is for memory allocations of > + * types (b) and (c). The convenience macro NOGC should be used where > + * possible. > + * > + * NOGC (and ctx->nogc_gc) may ONLY be used with functions which > + * explicitly declare that it''s OK. Use with nonconsenting functions > + * may result in leaks of those functions'' internal allocations on the > + * psuedo-gc. > */ > -/* register @ptr in @gc for free on exit from outermost libxl callframe. */ > -_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr); > +/* register ptr in gc for free on exit from outermost libxl callframe. */ > +_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr /* may be NULL */); > /* if this is the outermost libxl callframe then free all pointers in @gc */ > _hidden void libxl__free_all(libxl__gc *gc); > /* allocate and zero @bytes. (similar to a gc''d malloc(3)+memzero()) */ > @@ -2110,7 +2119,7 @@ _hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid); > #define GC_INIT(ctx) libxl__gc gc[1]; LIBXL_INIT_GC(gc[0],ctx) > #define GC_FREE libxl__free_all(gc) > #define CTX libxl__gc_owner(gc) > -#define NOGC NULL > +#define NOGC (&CTX->nogc_gc) /* pass only to consenting functions */ > > /* Allocation macros all of which use the gc. */ > > diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c > index 67ef82c..08c7dac 100644 > --- a/tools/libxl/libxl_utils.c > +++ b/tools/libxl/libxl_utils.c > @@ -58,8 +58,7 @@ char *libxl_domid_to_name(libxl_ctx *ctx, uint32_t domid) > char *libxl__domid_to_name(libxl__gc *gc, uint32_t domid) > { > char *s = libxl_domid_to_name(libxl__gc_owner(gc), domid); > - if ( s ) > - libxl__ptr_add(gc, s); > + libxl__ptr_add(gc, s); > return s; > } > > @@ -107,8 +106,7 @@ char *libxl_cpupoolid_to_name(libxl_ctx *ctx, uint32_t poolid) > char *libxl__cpupoolid_to_name(libxl__gc *gc, uint32_t poolid) > { > char *s = libxl_cpupoolid_to_name(libxl__gc_owner(gc), poolid); > - if ( s ) > - libxl__ptr_add(gc, s); > + libxl__ptr_add(gc, s); > return s; > } > > diff --git a/tools/libxl/libxl_xshelp.c b/tools/libxl/libxl_xshelp.c > index 993f527..7fdf164 100644 > --- a/tools/libxl/libxl_xshelp.c > +++ b/tools/libxl/libxl_xshelp.c > @@ -86,11 +86,8 @@ char * libxl__xs_read(libxl__gc *gc, xs_transaction_t t, const char *path) > char *ptr; > > ptr = xs_read(ctx->xsh, t, path, NULL); > - if (ptr != NULL) { > - libxl__ptr_add(gc, ptr); > - return ptr; > - } > - return 0; > + libxl__ptr_add(gc, ptr); > + return ptr; > } > > char *libxl__xs_get_dompath(libxl__gc *gc, uint32_t domid)
Ian Campbell
2012-Jun-13 13:25 UTC
Re: [PATCH 17/19] libxl: do not leak spawned middle children
On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:> libxl__spawn_spawn would, when libxl__spawn_detach was called, make > the spawn become idle immediately. However it still has a child > process which needs to be waited for: the `detachable'' spawned > child. > > This is wrong because the ultimate in-libxl caller may return to the > application, with a child process still forked but not reaped libxl > contrary to the documented behaviour of libxl. > > Instead, replace libxl__spawn_detach with libxl__spawn_initiate_detach > which is asynchronous. The detachable spawned children are abolished; > instead, we defer calling back to the in-libxl user until the middle > child has been reaped. > > Also, remove erroneous comment suggesting that `death'' callback > parameter to libxl__ev_child_fork may be NULL. It may not, and there > are no callers which pass NULL. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com> (couple of tiny queries below) [...]> diff --git a/tools/libxl/libxl_exec.c b/tools/libxl/libxl_exec.c > index cfa379c..bb85682 100644 > --- a/tools/libxl/libxl_exec.c > +++ b/tools/libxl/libxl_exec.c > @@ -238,15 +238,16 @@ err: > /* > * Full set of possible states of a libxl__spawn_state and its _detachable: > * > - * ss-> ss-> ss-> | ssd-> ssd-> > - * timeout xswatch ssd | mid ss > - * - Undefined undef undef no | - - > - * - Idle Idle Idle no | - - > - * - Active Active Active yes | Active yes > - * - Partial Active/Idle Active/Idle maybe | Active/Idle yes (if exists) > - * - Detached - - - | Active no > + * detaching failed mid timeout xswatch > + * - Undefined undef undef - undef undef > + * - Idle any any Idle Idle Idle > + * - Attached OK 0 0 Active Active Active > + * - Attached Failed 0 1 Active Idle Idle > + * - Detaching 1 maybe Active Idle Idle > + * - Partial any any Idle Active/Idle Active/Idle > * > - * When in state Detached, the middle process has been sent a SIGKILL. > + * When in states Detaching or Attached Failed, the middle process has > + * been sent a SIGKILL. > */ > > /* Event callbacks. */ > @@ -257,19 +258,18 @@ static void spawn_timeout(libxl__egc *egc, libxl__ev_time *ev, > static void spawn_middle_death(libxl__egc *egc, libxl__ev_child *childw, > pid_t pid, int status); > > -/* Precondition: Partial. Results: Detached. */ > +/* Precondition: Partial. Results: Idle. */ > static void spawn_cleanup(libxl__gc *gc, libxl__spawn_state *ss); > > -/* Precondition: Partial; caller has logged failure reason. > - * Results: Caller notified of failure; > - * after return, ss may be completely invalid as caller may reuse it */ > -static void spawn_failed(libxl__egc *egc, libxl__spawn_state *ss); > +/* Precondition: Attached or Detaching; caller has logged failure reason. > + * Results: Detaching, or Attached Failing */Is it Failing or Failed? You use Failed elsewhere. [...]> @@ -998,8 +997,8 @@ _hidden int libxl__device_pci_destroy_all(libxl__gc *gc, uint32_t domid); > * > * Higher-level double-fork and separate detach eg as for device models > * > - * Each libxl__spawn_state is in one of the conventional states > - * Undefined, Idle, Active > + * Each libxl__spawn_state is in one of these states > + * Undefined, Idle, Attached, Detaching"Attached OK" and "Attached Failed" as described above aren''t really full states, just sub-states? Ian.
Ian Campbell
2012-Jun-13 16:48 UTC
Re: [PATCH] libxl: further fixups re LIBXL_DOMAIN_TYPE process
On Mon, 2012-06-11 at 17:43 +0100, Ian Jackson wrote:> Here''s another patch which needs to go on top of my async save/restore > series. > > Ian. > > From: Ian Jackson <ian.jackson@eu.citrix.com> > Subject: [PATCH] libxl: further fixups re LIBXL_DOMAIN_TYPE > > * Abolish the macro LIBXL__DOMAIN_IS_TYPE which had incorrect error > handlng. At every call site, replace it with an open-coded call tohandling> libxl_domain_type and check against LIBXL_DOMAIN_TYPE_INVALID. > > * This involves adding an `out:'' to libxl_domain_unpause. > > * In libxl_domain_destroy and do_pci_add, do not `default: abort();'' > if the domain type cannot be found. Instead switch on > LIBXL_DOMAIN_TYPE_INVALID specifically and do some actual error > handling. > > * In libxl__primary_console_find, remove a spurious default clause > from the domain type switch. > > * In libxl_domain_suspend (as reorganised) error check, check for > LIBXL_DOMAIN_TYPE_INVALID and remove a pointless extra log message. > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson
2012-Jun-14 15:09 UTC
Re: [PATCH 02/19] libxl: domain save: rename variables etc.
Ian Campbell writes ("Re: [Xen-devel] [PATCH 02/19] libxl: domain save:
rename variables etc."):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > Preparatory work for making domain suspend asynchronous:
> [...]
> > No functional change whatsoever.
>
> Given that and a quick skim of the changes:
>
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Thanks.
> Although in general I find it easier to review patches which contain
> only one particular class of "mostly automatic" cleanup (e.g.
renames
> separate from "pointerization" separate from local variable
removal
> etc).
OK, I will split it up more next time I do something like this. I
thought I already had rather too many pre- and post-patches, but I''m
happy to make more of them in future.
Thanks,
Ian.
Ian Jackson
2012-Jun-14 15:11 UTC
Re: [PATCH 03/19] libxl: domain restore: reshuffle, preparing for ao
Ian Campbell writes ("Re: [Xen-devel] [PATCH 03/19] libxl: domain restore:
reshuffle, preparing for ao"):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > We are going to arrange that libxl, instead of calling
> > xc_domain_restore, calls a stub function which forks and execs a
> > helper program, so that restore can be asynchronous rather than
> > blocking the whole toolstack.
...> > * Move the contents of domain_restore into its correct place in the
> > domain creation sequence. We put it inside
> > domcreate_bootloader_done, which now either calls
>
> "either" but no "or"? I suppose the or case is call
> domcreate_rebuild_done directly?
Yes. I have completed this sentence.
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Thanks.
> I confirmed at a high level that the same blocks of code exist and are
> called in the same overall order, but I didn''t do a line by line
> comparison of the blocks in question, assuming they really are mostly
> motion and adjustments for the new context.
Right. I did some semi-mechanical checks to confirm that assertion,
after the last round of hideous merge conflict doom.
Ian.
Ian Jackson
2012-Jun-14 15:26 UTC
Re: [PATCH 04/19] libxl: domain save: API changes for asynchrony
Ian Campbell writes ("Re: [Xen-devel] [PATCH 04/19] libxl: domain save: API
changes for asynchrony"):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > Change the internal and external APIs for domain save (suspend) to be
> > capable of asynchronous operation. The implementation remains
> > synchronous. The interfaces surrounding device model saving are still
> > synchronous.
...> > * The `suspend_callback'' function passed to
libxl_domain_save is
> > never called by the existing implementation in libxl. Abolish it.
>
> Furthermore xl never passes one in either.
Right. I will update the commit message to note this too.
> > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> A few minor comments below, but otherwise looks good to me.
Thanks...
> > +static void remus_crashed_cb(libxl__egc *egc,
> > + libxl__domain_suspend_state *dss, int
rc)
>
> I''m not sure "crashed" is quite right here,
it''s finished for whatever
> reason which may not necessarily be a crash (going forward it should
> rarely be a crash, I think). It''s "stopped" or
"done" or something.
How about "remus_target_gone_cb" ?
> > +int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int
flags,
> > + const libxl_asyncop_how *ao_how)
> > + if (type < 0) {
> > + LOG(ERROR,"domain %"PRIu32": unable to
determine domain type", domid);
>
> libxl__domain_type now logs for you.
I will bring forward the relevant hunk from my domain type final
fixups patch.
> > int libxl_domain_pause(libxl_ctx *ctx, uint32_t domid)
> [...]
> > @@ -1903,10 +1915,27 @@ struct libxl__domain_create_state {
> >
> > /*----- Domain suspend (save) functions -----*/
> >
> > -_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t
domid, int fd,
> > - libxl_domain_type type,
> > - int live, int debug,
> > - const
libxl_domain_remus_info *r_info);
> > +/* calls callback when done */
>
> Which callback? dss->callback I guess.
What a confusing way to quote this. You''re referring to this function
of course:
> > +_hidden void libxl__domain_suspend(libxl__egc *egc,
> > + libxl__domain_suspend_state *dss);
Gripe gripe these comments before functions are inducing you to
top-post !
Anyway, yes. I will clarify this comment.
Ian.
Ian Jackson
2012-Jun-14 15:31 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process [and 4 more messages]
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: [Xen-devel] [PATCH v3 00/18] libxl: domain save/restore: run in
a separate process
In-Reply-To: <1339577986.24104.149.camel@zakaz.uk.xensource.com>
References: <1339176870-32652-1-git-send-email-ian.jackson@eu.citrix.com>
<1339577986.24104.149.camel@zakaz.uk.xensource.com>
X-Mailer: VM 8.1.0 under 23.2.1 (i486-pc-linux-gnu)
FCC: ~/mail/Outbound
--text follows this line--
Ian Campbell writes ("Re: [Xen-devel] [PATCH v3 00/18] libxl: domain
save/restore: run in a separate process"):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > This is v3 of my series to asyncify save/restore, rebased to current
> > tip, retested, and with all comments addressed.
>
> There''s quite a lot of combinations which need testing here (PV,
HVM,
> HVM w/ stub dm, old vs new qemu etc etc), which of those have you tried?
>
> I tried a simple localhost migrate of a PV guest and:
Thanks for looking at all this and for testing it. I thought I had
tested localhost migration, but my shell history reveals, now that it
is pointed out to me, that in my tests the migration receiver process
had been running the old version of libxl.
Ian Campbell writes ("Re: [Xen-devel] [PATCH v3 00/18] libxl: domain
save/restore: run in a separate process"):> The first zero here is restore_fd, I think. But I read in the comment in
> the helper:
> > + * The helper talks on stdin and stdout, in binary in machine
> > + * endianness. The helper speaks first, and only when it has
a
> > + * callback to make. It writes a 16-bit number being the
message
> > + * length, and then the message body.
>
> So restore_fd == stdin => running two protocols over the same fd?
This is indeed why it''s not working. I have repro''d the
failure and
my tree now has a fix in it.
Ian Campbell writes ("Re: [Xen-devel] [PATCH v3 00/18] libxl: domain
save/restore: run in a separate process"):> Oh, right, migrate-receive takes the migration fd on stdin doesn''t
it,
> so that''s where it comes from. I still suspect it is wrong. Might
need
> to dup the input onto a safe fd?
Indeed.
> BTW, since I''ve been ctrl-c''ing "xl migrate" a
bunch I noticed that we
> seem to leak an "xl migrate-receive" and the restore side helper
> process. Probably pre-existing but I thought it worth mentioning.
I''ll look into this.
Thanks,
Ian.
Ian Jackson
2012-Jun-14 15:39 UTC
Re: [PATCH v3 00/18] libxl: domain save/restore: run in a separate process [and 4 more messages]
Ian Jackson writes ("Re: [Xen-devel] [PATCH v3 00/18] libxl: domain
save/restore: run in a separate process [and 4 more
messages]"):> To: Ian Campbell <Ian.Campbell@citrix.com>
> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Uh, sorry about the duplicated header.
> Ian Campbell writes ("Re: [Xen-devel] [PATCH v3 00/18] libxl: domain
save/restore: run in a separate process"):
> > BTW, since I''ve been ctrl-c''ing "xl
migrate" a bunch I noticed that we
> > seem to leak an "xl migrate-receive" and the restore side
helper
> > process. Probably pre-existing but I thought it worth mentioning.
>
> I''ll look into this.
This seems better now, at the tip of my series, at least. If I ctrl-c
xl migrate running via ssh then all the receiver processes, and the
relevant domain, seem to get cleaned up.
If I do xl migrate not running via ssh, eg
xl migrate -s '''' debian.guest.osstest ''xl
migrate-recieve''
then both the sender and receiver get my ^C so we leak the domain. I
think that''s the expected behaviour.
Ian.
Ian Jackson
2012-Jun-14 15:47 UTC
Re: [PATCH 08/19] libxl: wait for qemu to acknowledge logdirty command
Ian Campbell writes ("Re: [Xen-devel] [PATCH 08/19] libxl: wait for qemu to
acknowledge logdirty command"):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > The current migration code in libxl instructs qemu to start or stop
> > logdirty, but it does not wait for an acknowledgement from qemu before
> > continuing. This might lead to memory corruption (!)
...> > +static void logdirty_init(libxl__logdirty_switch *lds)
> > +{
> > + lds->cmd_path = 0;
> > + libxl__ev_xswatch_init(&lds->watch);
> > + libxl__ev_time_init(&lds->timeout);
>
> I meant to mention this once before when reviewing some patch or other
> but I''m not sure I actually did, so at the risk of repeating
myself:
>
> This watch with timeout pattern seems to be reasonably common (in fact,
> I''m not sure we have any watches without a timeout). It might be a
> candidate for a specific helper routine in the future?
Perhaps. We should think about this. I''m not sure it''s
necessary
now. The benefit might be relatively small, as the callback function
would more complicated. Perhaps we should integrate the single xs
read which most of these callers also have.
> > +static void switch_logdirty_xswatch(libxl__egc *egc,
libxl__ev_xswatch *watch,
> > + const char *watch_path, const char
*event_path)
> > +{
...> > + out:
> > + /* rc < 0: error
> > + * rc == 0: ok, we are done
> > + * rc == +1: need to keep waiting
> > + */
> > + libxl__xs_transaction_abort(gc, &t);
> > +
> > + if (!rc) {
> > + switch_logdirty_done(egc,dss,1);
> > + } else if (rc < 0) {
> > + LOG(ERROR,"logdirty switch: response read failed
(rc=%d)",rc);
>
> Is it only "response read failed" which can cause us to end up
here,
> looks like the read, rm etc could do it?
Oh yes. I should fix that message. (All of these paths have already
logged something already, but another message probably doesn''t hurt.)
Thanks,
Ian.
Ian Jackson
2012-Jun-14 16:48 UTC
Re: [PATCH 05/19] libxl: domain save/restore: run in a separate process
Ian Campbell writes ("Re: [Xen-devel] [PATCH 05/19] libxl: domain
save/restore: run in a separate process"):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > libxenctrl expects to be able to simply run the save or restore
> > operation synchronously. This won''t work well in a process
which is
> > trying to handle multiple domains.
...> > diff --git a/.hgignore b/.hgignore
> > index 27d8f79..05304ea 100644
> > ^tools/libxl/tmp\..*$
> > +^tools/libxl/.*\.new$
>
> Our naming scheme for these sorts of temporary files is rather in
> consistent (including an existing use of .new), oh well...
Hmm, yes. I took my cue from libxl_list.h, which is immediately before
_libxl_save_msgs_helper.[ch] in the Makefile.
> > +libxl-save-helper: $(SAVE_HELPER_OBJS) libxenlight.so
> > + $(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS)
$(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
>
> The changelog says that libxl-save-helper doesn''t link libxenlight
but
> it is declared as a dependency here and CFLAGS_libxenlight is included
> in SAVE_HELPER_OBJS'' CFLAGS above.
This is a mistake in the Makefile, which I have fixed.
> > testidl: testidl.o libxlutil.so libxenlight.so
> > $(CC) $(LDFLAGS) -o $@ testidl.o libxlutil.so
$(LDLIBS_libxenlight) $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
> [...]
> > @@ -170,6 +184,7 @@ install: all
> > $(INSTALL_DIR) $(DESTDIR)$(BASH_COMPLETION_DIR)
> > $(INSTALL_DIR) $(DESTDIR)$(XEN_RUN_DIR)
> > $(INSTALL_PROG) xl $(DESTDIR)$(SBINDIR)
> > + $(INSTALL_PROG) libxl-save-helper $(DESTDIR)$(PRIVATE_BINDIR)
>
> This needs an INSTALL_DIR for $(DESTDIR)$(PRIVATE_BINDIR) to support
> "make -C tools/libxl DESTDIR=xxx install" else:
> install: cannot create regular file
`/tmp/tmplKa0Y7/usr/lib/xen/bin'': No such file or directory
OK. I guess I never do that and it''s a bit of a mystery why one
would without having done a more general install before, but it''s
clearly a correct change.
> > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> > index b48452c..fea0c94 100644
...> > int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
> > - uint32_t size, void *data)
...> > + libxl_ctx *ctx = CTX;
>
> Any reason you didn''t s/ctx/CTX/ in one of your preparatory
patches?
I didn''t want to do that unless it was necessary. There was already
enough else going on. I don''t think having a ctx rather than using
CTX is wrong, although it is slightly less in the modern idiom.
> > void libxl__xc_domain_save(libxl__egc *egc,
libxl__domain_suspend_state *dss,
> > unsigned long vm_generationid_addr)
> > {
> > STATE_AO_GC(dss->ao);
> > - int r;
> > + int r, rc;
> > + uint32_t toolstack_data_len;
> > +
> > + /* Resources we need to free */
> > + uint8_t *toolstack_data_buf = 0;
> > +
> > + unsigned cbflags = libxl__srm_callout_enumcallbacks_save
> > + (&dss->shs.callbacks.save.a);
> > +
> > + r = libxl__toolstack_save(dss->domid, &toolstack_data_buf,
> > + &toolstack_data_len, dss);
>
> This seems to be called twice? I''m looking at the source after the
full
> series is applied and I see
> dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
> in libxl__domain_suspend as well as this call here.
But in fact dss->shs.callbacks.save.toolstack_save is never used
anywhere in that version. The bug is that libxl__xc_domain_save
should call the callback function (if it is non-null), not call
libxl__toolstack_save directly.
> The callback one seems to be otherwise unused.
Exactly.
I have fixed this:
* libxl__xc_domain_save uses supplied callback function pointer,
rather than calling libxl__toolstack_save directly;
toolstack data save callback is only supplied to libxc if
in-libxl caller supplied a callback.
> > +static void run_helper(libxl__egc *egc, libxl__save_helper_state
*shs,
> > + const char *mode_arg, int data_fd,
> > + const unsigned long *argnums, int num_argnums)
> > +{
...> > + libxl__carefd_begin();
> > + for (i=0; i<2; i++) {
> > + int fds[2];
> > + if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; }
> > + childs_pipes[i] = libxl__carefd_record(CTX, fds[i]);
> > + shs->pipes[i] = libxl__carefd_record(CTX, fds[!i]);
>
> Using !i here is clever, but I had to deploy a pen to be sure the
> assignments were all correct.
>
> Open coding this simple loop would also give you the opportunity the
> have comments like "/* This is the helper processes''s stdout
*/" etc in
> the appropriate place to aid in comprehension when checking the correct
> end of each pipe goes in the right place.
Perhaps the right answer is to use a better variable name than `i''.
I wrote it out longhand and it looked like this:
libxl__carefd_begin();
int fds[2];
/* child''s stdin */
if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; }
childs_pipes[0] = libxl__carefd_record(CTX, fds[0]);
shs->pipes[0] = libxl__carefd_record(CTX, fds[1]);
/* child''s stdout */
if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; }
childs_pipes[1] = libxl__carefd_record(CTX, fds[1]);
shs->pipes[1] = libxl__carefd_record(CTX, fds[0]);
libxl__carefd_unlock();
which is a great way to obscure the subtle differences between the two
stanzas. How about:
libxl__carefd_begin();
int childfd;
for (childfd=0; childfd<2; childfd++) {
/* Setting up the pipe for the child''s fd childfd */
int fds[2];
if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; }
int childs_end = childfd==0 ? 0 /*read*/ : 1 /*write*/;
int our_end = childfd==0 ? 1 /*write*/ : 0 /*read*/;
childs_pipes[childfd] = libxl__carefd_record(CTX, fds[childs_end]);
shs->pipes[childfd] = libxl__carefd_record(CTX, fds[our_end]);
}
libxl__carefd_unlock();
?
> > +static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
> > + int fd, short events, short
revents)
> > +{
> > + libxl__save_helper_state *shs = CONTAINER_OF(ev, *shs, readable);
> > + STATE_AO_GC(shs->ao);
> > + int rc, errnoval;
> > +
> > + if (revents & POLLPRI) {
> > + LOG(ERROR, "%s signaled POLLERR",
shs->stdout_what);
>
> signalled ?
>
> (it''s not impossible that my spell checker is wrong, google seem
to
> think both are ok)
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/kill.html
uses "signaled".
> The if says POLLPRI but the log says POLLERR?
This should check and log both.
> > + rc = ERROR_FAIL;
> > + out:
> > + /* this is here because otherwise we bypass the decl of msg[]
*/
>
> I don''t get this comment, why can''t "out:" be
in the normal place after
> the non-error return?
Because it is not legal to "goto" within the scope of a variable whose
size is not a constant. The alternative would be to introduce a block
scope starting before `unsigned char msg[msglen]'' and ending after
`return''.
Arguably msg[msglen] is asking too much (up to 64K) of the caller''s
stack. Should I change it ?
> > + if (!shs->completed) {
> > + if (!shs->rc)
> > + LOG(ERROR,"%s exited without signaling
completion",what);
>
> signalling again, same caveat
I''ll go with what SuS says.
> > --- /dev/null
> > +++ b/tools/libxl/libxl_save_helper.c
> > @@ -0,0 +1,279 @@
> > +/*
> > + * Copyright (C) 2012 Citrix Ltd.
> > + *
> > + * This program is free software; you can redistribute it and/or
modify
> > + * it under the terms of the GNU Lesser General Public License as
published
> > + * by the Free Software Foundation; version 2.1 only. with the
special
> > + * exception on linking described in file LICENSE.
>
> LGPL, or perhaps GPL since this is a helper?
>
> (I suppose the same argument could be made for xl itself)
I can see no reason to deviate from the licence of the rest of libxl.
If nothing else, code may move between the helper and libxl proper.
> > +int helper_getreply(void *user)
> > +{
> > + int v;
> > + int r = read_exactly(0, &v, sizeof(v));
> > + if (r<=0) exit(-2);
>
> You use -1 a lot but here you use -2, are we supposed to be able to
> infer something from the specific error code?
Not really. I used -1 for general system call failures. This
particular situation might include a protocol error by the parent,
though, so I thought I''d distinguish it. That way if you get a
message saying the helper exited with a nonzero exit status foobar you
get slightly more information.
I could make this more formal and be more consistent with these exit
statuses, or alternatively I could abolish it. I don''t think
it''s
critical - nothing turns on this exit status.
> > diff --git a/tools/libxl/libxl_save_msgs_gen.pl
b/tools/libxl/libxl_save_msgs_gen.pl
> > new file mode 100755
> > index 0000000..cd0837e
> > --- /dev/null
> > +++ b/tools/libxl/libxl_save_msgs_gen.pl
> > @@ -0,0 +1,393 @@
> > +#!/usr/bin/perl -w
> > +
> > +use warnings;
> > +use strict;
> > +use POSIX;
> > +
> > +our $debug = 0; # produce copius debugging output at run-time?
>
> copious
>
> (which is probably the most useful feedback you are going to get from me
> on a Perl script ;-))
Fixed.
> > +foreach my $ah (qw(callout helper)) {
> > + $out_body{$ah} .= <<END.($ah eq ''callout''
? <<END : <<END);
> [...]
> > +END
> [...]
> > +END
> [...]
> > +END
>
> I think I can infer what this does, but wow ;-)
Following a suggestion from Mark Wooding I have changed this to:
<<END_BOTH.($ah eq ''callout'' ?
<<END_CALLOUT : <<END_HELPER);
#include "libxl_osdeps.h"
#include <assert.h>
#include <string.h>
#include <stdint.h>
#include <limits.h>
END_BOTH
#include "libxl_internal.h"
END_CALLOUT
#include "_libxl_save_msgs_${ah}.h"
#include <xenctrl.h>
#include <xenguest.h>
END_HELPER
which I think is much clearer.
Ian.
Ian Jackson
2012-Jun-14 16:53 UTC
Re: [PATCH 07/19] libxl: provide libxl__xs_*_checked and libxl__xs_transaction_*
Ian Campbell writes ("Re: [Xen-devel] [PATCH 07/19] libxl: provide
libxl__xs_*_checked and libxl__xs_transaction_*"):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > These useful utility functions make dealing with xenstore a little
> > less painful.
> >
> > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Thanks.
> (minor observation and a typo below)
...> > +int libxl__xs_write_checked(libxl__gc *gc, xs_transaction_t t,
> > + const char *path, const char *string);
>
> I suppose in the future we might consider merging this with
> libxl__xs_write -- there''s no reason not to always log a failed
write,
> is there?
Maybe. It might fail due to EPERM and there might be another strategy
the caller might have available. Although this is rare enough that
perhaps such callers should use raw xs calls.
Arguably the same applies to read.
> > + * 0 commited successfully
>
> committed
Fixed.
Ian.
Ian Jackson
2012-Jun-14 16:58 UTC
Re: [PATCH 14/19] libxl: Get compiler to warn about gc_opt==NULL
Ian Campbell writes ("Re: [Xen-devel] [PATCH 14/19] libxl: Get compiler to
warn about gc_opt==NULL"):> On Wed, 2012-06-13 at 14:08 +0100, Ian Campbell wrote:
> > This patch will itself break the build until a subsequent patch,
won''t
> > it? Shuffle it afterwards?
>
> Nevermind, I accidentally skipped #13...
:-).
> > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> >
> > Wherever it ends up in the sequence:
> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
Thanks,
Ian.
Ian Jackson
2012-Jun-14 17:08 UTC
Re: [PATCH 17/19] libxl: do not leak spawned middle children
Ian Campbell writes ("Re: [Xen-devel] [PATCH 17/19] libxl: do not leak
spawned middle children"):> On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote:
> > Also, remove erroneous comment suggesting that `death''
callback
> > parameter to libxl__ev_child_fork may be NULL. It may not, and there
> > are no callers which pass NULL.
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Thanks.
> > -static void spawn_failed(libxl__egc *egc, libxl__spawn_state *ss);
> > +/* Precondition: Attached or Detaching; caller has logged failure
reason.
> > + * Results: Detaching, or Attached Failing */
>
> Is it Failing or Failed? You use Failed elsewhere.
`Failed'' will do. Fixed.
> [...]
> > @@ -998,8 +997,8 @@ _hidden int
libxl__device_pci_destroy_all(libxl__gc *gc, uint32_t domid);
> > *
> > * Higher-level double-fork and separate detach eg as for device
models
> > *
> > - * Each libxl__spawn_state is in one of the conventional states
> > - * Undefined, Idle, Active
> > + * Each libxl__spawn_state is in one of these states
> > + * Undefined, Idle, Attached, Detaching
>
> "Attached OK" and "Attached Failed" as described above
aren''t really
> full states, just sub-states?
Yes. I have clarified this:
* The difference between Attached OK and Attached Failed is not
* directly visible to callers - callers see these two the same,
* although of course Attached OK will hopefully eventually result in
* a call to detached_cb, whereas Attached Failed will end up
* in a call to failure_cb.
Ian.
Ian Campbell
2012-Jun-19 10:04 UTC
Re: [PATCH 04/19] libxl: domain save: API changes for asynchrony
On Thu, 2012-06-14 at 16:26 +0100, Ian Jackson wrote:> Ian Campbell writes ("Re: [Xen-devel] [PATCH 04/19] libxl: domain save: API changes for asynchrony"): > > On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote: > > > Change the internal and external APIs for domain save (suspend) to be > > > capable of asynchronous operation. The implementation remains > > > synchronous. The interfaces surrounding device model saving are still > > > synchronous. > ... > > > * The `suspend_callback'' function passed to libxl_domain_save is > > > never called by the existing implementation in libxl. Abolish it. > > > > Furthermore xl never passes one in either. > > Right. I will update the commit message to note this too. > > > > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> > > > > A few minor comments below, but otherwise looks good to me. > > Thanks... > > > > +static void remus_crashed_cb(libxl__egc *egc, > > > + libxl__domain_suspend_state *dss, int rc) > > > > I''m not sure "crashed" is quite right here, it''s finished for whatever > > reason which may not necessarily be a crash (going forward it should > > rarely be a crash, I think). It''s "stopped" or "done" or something. > > How about "remus_target_gone_cb" ?Sounds fine to me, unless you want to work the word "failover" into it?> > > int libxl_domain_pause(libxl_ctx *ctx, uint32_t domid) > > [...] > > > @@ -1903,10 +1915,27 @@ struct libxl__domain_create_state { > > > > > > /*----- Domain suspend (save) functions -----*/ > > > > > > -_hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, > > > - libxl_domain_type type, > > > - int live, int debug, > > > - const libxl_domain_remus_info *r_info); > > > +/* calls callback when done */ > > > > Which callback? dss->callback I guess. > > What a confusing way to quote this.Yes, sorry.> You''re referring to this function > of course: > > > > +_hidden void libxl__domain_suspend(libxl__egc *egc, > > > + libxl__domain_suspend_state *dss); > > Gripe gripe these comments before functions are inducing you to > top-post ! > > Anyway, yes. I will clarify this comment.Thanks. Ian.
Ian Jackson
2012-Jun-19 13:02 UTC
Re: [PATCH 04/19] libxl: domain save: API changes for asynchrony
Ian Campbell writes ("Re: [Xen-devel] [PATCH 04/19] libxl: domain save: API
changes for asynchrony"):> On Thu, 2012-06-14 at 16:26 +0100, Ian Jackson wrote:
> > How about "remus_target_gone_cb" ?
>
> Sounds fine to me, unless you want to work the word "failover"
into it?
remus_failover_cb ?
Ian.
Ian Campbell
2012-Jun-19 13:33 UTC
Re: [PATCH 08/19] libxl: wait for qemu to acknowledge logdirty command
On Thu, 2012-06-14 at 16:47 +0100, Ian Jackson wrote:> Ian Campbell writes ("Re: [Xen-devel] [PATCH 08/19] libxl: wait for qemu to acknowledge logdirty command"): > > On Fri, 2012-06-08 at 18:34 +0100, Ian Jackson wrote: > > > The current migration code in libxl instructs qemu to start or stop > > > logdirty, but it does not wait for an acknowledgement from qemu before > > > continuing. This might lead to memory corruption (!) > ... > > > +static void logdirty_init(libxl__logdirty_switch *lds) > > > +{ > > > + lds->cmd_path = 0; > > > + libxl__ev_xswatch_init(&lds->watch); > > > + libxl__ev_time_init(&lds->timeout); > > > > I meant to mention this once before when reviewing some patch or other > > but I''m not sure I actually did, so at the risk of repeating myself: > > > > This watch with timeout pattern seems to be reasonably common (in fact, > > I''m not sure we have any watches without a timeout). It might be a > > candidate for a specific helper routine in the future? > > Perhaps. We should think about this.Yes, I should have said the phrase "4.3" in there somewhere.> I''m not sure it''s necessary > now. The benefit might be relatively small, as the callback function > would more complicated.Could still have two callbacks, and just helpers for the setup & teardown phases?> Perhaps we should integrate the single xs read which most of these callers also have.You mean pass in the value of the watched key? That''s also a possibility.> > > > +static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch, > > > + const char *watch_path, const char *event_path) > > > +{ > ... > > > + out: > > > + /* rc < 0: error > > > + * rc == 0: ok, we are done > > > + * rc == +1: need to keep waiting > > > + */ > > > + libxl__xs_transaction_abort(gc, &t); > > > + > > > + if (!rc) { > > > + switch_logdirty_done(egc,dss,1); > > > + } else if (rc < 0) { > > > + LOG(ERROR,"logdirty switch: response read failed (rc=%d)",rc); > > > > Is it only "response read failed" which can cause us to end up here, > > looks like the read, rm etc could do it? > > Oh yes. I should fix that message. (All of these paths have already > logged something already, but another message probably doesn''t hurt.) > > Thanks, > Ian.
Ian Campbell
2012-Jun-19 13:50 UTC
Re: [PATCH 05/19] libxl: domain save/restore: run in a separate process
> > > testidl: testidl.o libxlutil.so libxenlight.so > > > $(CC) $(LDFLAGS) -o $@ testidl.o libxlutil.so $(LDLIBS_libxenlight) $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) > > [...] > > > @@ -170,6 +184,7 @@ install: all > > > $(INSTALL_DIR) $(DESTDIR)$(BASH_COMPLETION_DIR) > > > $(INSTALL_DIR) $(DESTDIR)$(XEN_RUN_DIR) > > > $(INSTALL_PROG) xl $(DESTDIR)$(SBINDIR) > > > + $(INSTALL_PROG) libxl-save-helper $(DESTDIR)$(PRIVATE_BINDIR) > > > > This needs an INSTALL_DIR for $(DESTDIR)$(PRIVATE_BINDIR) to support > > "make -C tools/libxl DESTDIR=xxx install" else: > > install: cannot create regular file `/tmp/tmplKa0Y7/usr/lib/xen/bin'': No such file or directory > > OK. I guess I never do that and it''s a bit of a mystery why one > would without having done a more general install before, but it''s > clearly a correct change.I build and install on a different machine, which means I install into DESTDIR=`mkdtemp ...` and tar, copy etc. I do one full build to seed everything and then incrementally just the dir I''m working in, so this hits me because on the incremental ones the fresh DESTDIR doesn''t have the directory yet.> > > > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > > > index b48452c..fea0c94 100644 > ... > > > int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf, > > > - uint32_t size, void *data) > ... > > > + libxl_ctx *ctx = CTX; > > > > Any reason you didn''t s/ctx/CTX/ in one of your preparatory patches? > > I didn''t want to do that unless it was necessary. There was already > enough else going on. I don''t think having a ctx rather than using > CTX is wrong, although it is slightly less in the modern idiom.Fair enough.> > > > void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss, > > > unsigned long vm_generationid_addr) > > > { > > > STATE_AO_GC(dss->ao); > > > - int r; > > > + int r, rc; > > > + uint32_t toolstack_data_len; > > > + > > > + /* Resources we need to free */ > > > + uint8_t *toolstack_data_buf = 0; > > > + > > > + unsigned cbflags = libxl__srm_callout_enumcallbacks_save > > > + (&dss->shs.callbacks.save.a); > > > + > > > + r = libxl__toolstack_save(dss->domid, &toolstack_data_buf, > > > + &toolstack_data_len, dss); > > > > This seems to be called twice? I''m looking at the source after the full > > series is applied and I see > > dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save; > > in libxl__domain_suspend as well as this call here. > > But in fact dss->shs.callbacks.save.toolstack_save is never used > anywhere in that version. The bug is that libxl__xc_domain_save > should call the callback function (if it is non-null), not call > libxl__toolstack_save directly. > > > The callback one seems to be otherwise unused. > > Exactly. > > I have fixed this: > > * libxl__xc_domain_save uses supplied callback function pointer,^ the ?> rather than calling libxl__toolstack_save directly; > toolstack data save callback is only supplied to libxc if > in-libxl caller supplied a callback.^ "the" or "an"? Otherwise looks good.> > > +static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs, > > > + const char *mode_arg, int data_fd, > > > + const unsigned long *argnums, int num_argnums) > > > +{ > ... > > > + libxl__carefd_begin(); > > > + for (i=0; i<2; i++) { > > > + int fds[2]; > > > + if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; } > > > + childs_pipes[i] = libxl__carefd_record(CTX, fds[i]); > > > + shs->pipes[i] = libxl__carefd_record(CTX, fds[!i]); > > > > Using !i here is clever, but I had to deploy a pen to be sure the > > assignments were all correct. > > > > Open coding this simple loop would also give you the opportunity the > > have comments like "/* This is the helper processes''s stdout */" etc in > > the appropriate place to aid in comprehension when checking the correct > > end of each pipe goes in the right place. > > Perhaps the right answer is to use a better variable name than `i''. > I wrote it out longhand and it looked like this:[...]> which is a great way to obscure the subtle differences between the two > stanzas. How about: > > libxl__carefd_begin(); > int childfd; > for (childfd=0; childfd<2; childfd++) { > /* Setting up the pipe for the child''s fd childfd */ > int fds[2]; > if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; } > int childs_end = childfd==0 ? 0 /*read*/ : 1 /*write*/; > int our_end = childfd==0 ? 1 /*write*/ : 0 /*read*/; > childs_pipes[childfd] = libxl__carefd_record(CTX, fds[childs_end]); > shs->pipes[childfd] = libxl__carefd_record(CTX, fds[our_end]); > } > libxl__carefd_unlock(); > > ?Yes renaming that variable, adding those inline /*read*/ etc comments and the local variable naming has really helped, thanks> > > + rc = ERROR_FAIL; > > > + out: > > > + /* this is here because otherwise we bypass the decl of msg[] */ > > > > I don''t get this comment, why can''t "out:" be in the normal place after > > the non-error return? > > Because it is not legal to "goto" within the scope of a variable whose > size is not a constant.Ah OK, I didn''t realise this, another interesting C quirk!> The alternative would be to introduce a block > scope starting before `unsigned char msg[msglen]'' and ending after > `return''. > > Arguably msg[msglen] is asking too much (up to 64K) of the caller''s > stack. Should I change it ?I think it''s OK in a userspace process. Although one thing to watch out for would be the stack guard page which modern Linux has -- this can cause issues with large stack allocations since you trigger the guard page instead of growing the stack like you''d expect -- I suppose this is really a bug in your compiler and/or libc but it does seem common enough to avoid for now...> > > --- /dev/null > > > +++ b/tools/libxl/libxl_save_helper.c > > > @@ -0,0 +1,279 @@ > > > +/* > > > + * Copyright (C) 2012 Citrix Ltd. > > > + * > > > + * This program is free software; you can redistribute it and/or modify > > > + * it under the terms of the GNU Lesser General Public License as published > > > + * by the Free Software Foundation; version 2.1 only. with the special > > > + * exception on linking described in file LICENSE. > > > > LGPL, or perhaps GPL since this is a helper? > > > > (I suppose the same argument could be made for xl itself) > > I can see no reason to deviate from the licence of the rest of libxl. > If nothing else, code may move between the helper and libxl proper.That''s a very good point.> > > > +int helper_getreply(void *user) > > > +{ > > > + int v; > > > + int r = read_exactly(0, &v, sizeof(v)); > > > + if (r<=0) exit(-2); > > > > You use -1 a lot but here you use -2, are we supposed to be able to > > infer something from the specific error code? > > Not really. I used -1 for general system call failures. This > particular situation might include a protocol error by the parent, > though, so I thought I''d distinguish it. That way if you get a > message saying the helper exited with a nonzero exit status foobar you > get slightly more information. > > I could make this more formal and be more consistent with these exit > statuses, or alternatively I could abolish it. I don''t think it''s > critical - nothing turns on this exit status.Lets just leave it then.> > > +foreach my $ah (qw(callout helper)) { > > > + $out_body{$ah} .= <<END.($ah eq ''callout'' ? <<END : <<END); > > [...] > > > +END > > [...] > > > +END > > [...] > > > +END > > > > I think I can infer what this does, but wow ;-) > > Following a suggestion from Mark Wooding I have changed this to: > > <<END_BOTH.($ah eq ''callout'' ? <<END_CALLOUT : <<END_HELPER); > #include "libxl_osdeps.h" > > #include <assert.h> > #include <string.h> > #include <stdint.h> > #include <limits.h> > END_BOTH > > #include "libxl_internal.h" > > END_CALLOUT > > #include "_libxl_save_msgs_${ah}.h" > #include <xenctrl.h> > #include <xenguest.h> > > END_HELPER > > which I think is much clearer.I agree with Mark, thanks!> > Ian.
Ian Campbell
2012-Jun-19 15:15 UTC
Re: [PATCH 04/19] libxl: domain save: API changes for asynchrony
On Tue, 2012-06-19 at 14:02 +0100, Ian Jackson wrote:> Ian Campbell writes ("Re: [Xen-devel] [PATCH 04/19] libxl: domain save: API changes for asynchrony"): > > On Thu, 2012-06-14 at 16:26 +0100, Ian Jackson wrote: > > > How about "remus_target_gone_cb" ? > > > > Sounds fine to me, unless you want to work the word "failover" into it? > > remus_failover_cb ?Sounds good.