Stefano Stabellini
2012-Jan-19 12:12 UTC
[PATCH 0/3] libxl: save additional info to the qemu chunk
Hi all, this patch series introduces an optional callback argument to xc_domain_restore that can be used by toolstacks to restore the qemu state (and whatever else is present in the qemu chunk) as they see fit. The callback is used by libxl to restore some additional information from the qemu chunk (saving the qemu chunk is already fully handled by libxl). Stefano Stabellini (3): libxc: add a callback to xc_domain_restore libxl: extract the qemu state file from the save image libxl: introduce QEMU_HEADER tools/libxc/xc_domain_restore.c | 19 ++++-- tools/libxc/xenguest.h | 17 ++++- tools/libxl/libxl_dm.c | 5 ++ tools/libxl/libxl_dom.c | 142 +++++++++++++++++++++++++++++++++++++- tools/libxl/libxl_internal.h | 2 + tools/xcutils/xc_restore.c | 3 +- 6 files changed, 175 insertions(+), 13 deletions(-) Cheers, Stefano
Stefano Stabellini
2012-Jan-19 12:12 UTC
[PATCH 1/3] libxc: add a callback to xc_domain_restore
Introduce a callback argument to xc_domain_restore, so that the caller can restore the Qemu state file by itself. If the callback is NULL, libxc will take care of dumping the qemu state file as usual. Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> --- tools/libxc/xc_domain_restore.c | 19 ++++++++++++++----- tools/libxc/xenguest.h | 17 ++++++++++++++--- tools/libxl/libxl_dom.c | 2 +- tools/xcutils/xc_restore.c | 3 ++- 4 files changed, 31 insertions(+), 10 deletions(-) diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c index 14451d1..fdc3483 100644 --- a/tools/libxc/xc_domain_restore.c +++ b/tools/libxc/xc_domain_restore.c @@ -1248,7 +1248,8 @@ static int apply_batch(xc_interface *xch, uint32_t dom, struct restore_ctx *ctx, int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, unsigned int store_evtchn, unsigned long *store_mfn, unsigned int console_evtchn, unsigned long *console_mfn, - unsigned int hvm, unsigned int pae, int superpages) + unsigned int hvm, unsigned int pae, int superpages, + struct restore_callbacks* callbacks) { DECLARE_DOMCTL; int rc = 1, frc, i, j, n, m, pae_extended_cr3 = 0, ext_vcpucontext = 0; @@ -1976,10 +1977,18 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, goto out; finish_hvm: - /* Dump the QEMU state to a state file for QEMU to load */ - if ( dump_qemu(xch, dom, &tailbuf.u.hvm) ) { - PERROR("Error dumping QEMU state to file"); - goto out; + if (callbacks == NULL || callbacks->extract_qemu_savestate == NULL) { + /* Dump the QEMU state to a state file for QEMU to load */ + if ( dump_qemu(xch, dom, &tailbuf.u.hvm) ) { + PERROR("Error dumping QEMU state to file"); + goto out; + } + } else { + if ( callbacks->extract_qemu_savestate(tailbuf.u.hvm.qemubuf, + tailbuf.u.hvm.qemubufsize, callbacks->data) ) { + PERROR("Error calling extract_qemu_savestate"); + goto out; + } } /* These comms pages need to be zeroed at the start of day */ diff --git a/tools/libxc/xenguest.h b/tools/libxc/xenguest.h index 4475ee9..9447412 100644 --- a/tools/libxc/xenguest.h +++ b/tools/libxc/xenguest.h @@ -61,6 +61,15 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter struct save_callbacks* callbacks, int hvm); +/* callbacks provided by xc_domain_restore */ +struct restore_callbacks { + /* callback to extract the qemu save state */ + int (*extract_qemu_savestate)(uint8_t *buf, uint32_t size, void* data); + + /* to be provided as the last argument to each callback function */ + void* data; +}; + /** * This function will restore a saved domain. * @@ -72,15 +81,17 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter * @parm hvm non-zero if this is a HVM restore * @parm pae non-zero if this HVM domain has PAE support enabled * @parm superpages non-zero to allocate guest memory with superpages + * @parm callbacks non-NULL to receive a callback to extract the qemu state * @return 0 on success, -1 on failure */ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, unsigned int store_evtchn, unsigned long *store_mfn, unsigned int console_evtchn, unsigned long *console_mfn, - unsigned int hvm, unsigned int pae, int superpages); + unsigned int hvm, unsigned int pae, int superpages, + struct restore_callbacks *callbacks); /** - * xc_domain_restore writes a file to disk that contains the device - * model saved state. + * If callbacks is NULL, xc_domain_restore writes a file to disk that + * contains the device model saved state. * The pathname of this file is XC_DEVICE_MODEL_RESTORE_FILE; The domid * of the new domain is automatically appended to the filename, * separated by a ".". diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c index c898d89..fd2b051 100644 --- a/tools/libxl/libxl_dom.c +++ b/tools/libxl/libxl_dom.c @@ -373,7 +373,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, rc = xc_domain_restore(ctx->xch, fd, domid, state->store_port, &state->store_mfn, state->console_port, &state->console_mfn, - hvm, pae, superpages); + hvm, pae, superpages, NULL); if ( rc ) { LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain"); return ERROR_FAIL; diff --git a/tools/xcutils/xc_restore.c b/tools/xcutils/xc_restore.c index 985cbec..eb1d31c 100644 --- a/tools/xcutils/xc_restore.c +++ b/tools/xcutils/xc_restore.c @@ -46,7 +46,8 @@ main(int argc, char **argv) superpages = !!hvm; ret = xc_domain_restore(xch, io_fd, domid, store_evtchn, &store_mfn, - console_evtchn, &console_mfn, hvm, pae, superpages); + console_evtchn, &console_mfn, hvm, pae, superpages, + NULL); if ( ret == 0 ) { -- 1.7.2.5
Stefano Stabellini
2012-Jan-19 12:13 UTC
[PATCH 2/3] libxl: extract the qemu state file from the save image
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> --- tools/libxl/libxl_dm.c | 5 +++++ tools/libxl/libxl_dom.c | 42 +++++++++++++++++++++++++++++++++++++++++- tools/libxl/libxl_internal.h | 1 + 3 files changed, 47 insertions(+), 1 deletions(-) diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c index 97d91b4..3f19150 100644 --- a/tools/libxl/libxl_dm.c +++ b/tools/libxl/libxl_dm.c @@ -41,6 +41,11 @@ const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid) return libxl__sprintf(gc, "/var/lib/xen/qemu-save.%d", domid); } +const char *libxl__device_model_restorefile(libxl__gc *gc, uint32_t domid) +{ + return libxl__sprintf(gc, "/var/lib/xen/qemu-resume.%d", domid); +} + const char *libxl__domain_device_model(libxl__gc *gc, libxl_device_model_info *info) { diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c index fd2b051..f33e572 100644 --- a/tools/libxl/libxl_dom.c +++ b/tools/libxl/libxl_dom.c @@ -347,6 +347,40 @@ out: return rc; } +struct restore_callbacks_arg { + libxl__gc *gc; + uint32_t domid; +}; + +static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, + void *data) +{ + struct restore_callbacks_arg *arg = data; + libxl__gc *gc = (libxl__gc *)arg->gc; + libxl_ctx *ctx = libxl__gc_owner(gc); + uint32_t domid = arg->domid; + int fd2 = -1, ret = 0; + const char *filename; + + filename = libxl__device_model_restorefile(gc, domid); + fd2 = open(filename, O_WRONLY|O_CREAT); + if (fd2 < 0) { + ret = fd2; + goto out; + } + + ret = libxl_write_exactly( + ctx, fd2, buf, size, "saved-state file", "qemu state"); + if (ret) + goto out; + ret = 0; + +out: + if (fd2 > 0) + close(fd2); + return ret; +} + int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, libxl_domain_build_info *info, libxl__domain_build_state *state, @@ -356,11 +390,17 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, /* read signature */ int rc; int hvm, pae, superpages; + struct restore_callbacks callbacks; + struct restore_callbacks_arg arg; switch (info->type) { case LIBXL_DOMAIN_TYPE_HVM: hvm = 1; superpages = 1; pae = info->u.hvm.pae; + arg.domid = domid; + arg.gc = gc; + callbacks.extract_qemu_savestate = libxl__domain_restore_device_model; + callbacks.data = &arg; break; case LIBXL_DOMAIN_TYPE_PV: hvm = 0; @@ -373,7 +413,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, rc = xc_domain_restore(ctx->xch, fd, domid, state->store_port, &state->store_mfn, state->console_port, &state->console_mfn, - hvm, pae, superpages, NULL); + hvm, pae, superpages, &callbacks); if ( rc ) { LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain"); return ERROR_FAIL; diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index fa7fb16..7f7578a 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -247,6 +247,7 @@ _hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, libxl_domain_type type, int live, int debug); _hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid); +_hidden const char *libxl__device_model_restorefile(libxl__gc *gc, uint32_t domid); _hidden int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd); _hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid); -- 1.7.2.5
Introduce a new QEMU_HEADER stored in the Qemu chunk right after the QEMU_SIGNATURE and record length, before the Qemu state, to preserve the physmap informations written by Qemu on xenstore. Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> --- tools/libxl/libxl_dom.c | 104 ++++++++++++++++++++++++++++++++++++++++-- tools/libxl/libxl_internal.h | 1 + 2 files changed, 100 insertions(+), 5 deletions(-) diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c index f33e572..0f3d0c3 100644 --- a/tools/libxl/libxl_dom.c +++ b/tools/libxl/libxl_dom.c @@ -359,9 +359,47 @@ static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, libxl__gc *gc = (libxl__gc *)arg->gc; libxl_ctx *ctx = libxl__gc_owner(gc); uint32_t domid = arg->domid; - int fd2 = -1, ret = 0; + int fd2 = -1, ret = 0, i = 0; const char *filename; + uint8_t *ptr = buf; + uint32_t count = 0; + uint64_t phys_offset_v = 0, start_addr_v = 0, size_v = 0; + uint32_t header_size = 0; + + if (strncmp((char *)ptr, QEMU_HEADER, size)) + goto no_header; + + ptr += sizeof(QEMU_HEADER); + memcpy(&count, ptr, sizeof(count)); + ptr += sizeof(count); + header_size = sizeof(QEMU_HEADER) + sizeof(count) + count * sizeof(uint64_t) * 3; + size -= header_size; + + for (i = 0; i < count; i++) { + memcpy(&phys_offset_v, ptr, sizeof(uint64_t)); + ptr += sizeof(uint64_t); + memcpy(&start_addr_v, ptr, sizeof(uint64_t)); + ptr += sizeof(uint64_t); + memcpy(&size_v, ptr, sizeof(uint64_t)); + ptr += sizeof(uint64_t); + + ret = libxl__xs_write(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%"PRIx64"/start_addr", + domid, phys_offset_v), "%"PRIx64, start_addr_v); + if (ret) { + ptr = buf + header_size; + break; + } + ret = libxl__xs_write(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%"PRIx64"/size", + domid, phys_offset_v), "%"PRIx64, size_v); + if (ret) { + ptr = buf + header_size; + break; + } + } +no_header: filename = libxl__device_model_restorefile(gc, domid); fd2 = open(filename, O_WRONLY|O_CREAT); if (fd2 < 0) { @@ -370,7 +408,7 @@ static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, } ret = libxl_write_exactly( - ctx, fd2, buf, size, "saved-state file", "qemu state"); + ctx, fd2, ptr, size, "saved-state file", "qemu state"); if (ret) goto out; ret = 0; @@ -642,11 +680,61 @@ out: return rc; } +static int libxl__domain_save_qemu_header(libxl__gc *gc, uint32_t domid, + int fd, char **buf) +{ + char *start_addr = NULL, *size = NULL, *phys_offset = NULL; + int len = 0, i = 0; + unsigned int num = 0; + uint32_t count = 0; + char *ptr = NULL, **entries = NULL; + uint64_t val = 0; + + entries = libxl__xs_directory(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap", domid), &num); + count = num; + + len = sizeof(QEMU_HEADER) + sizeof(count) + count * (sizeof(val) * 3); + *buf = libxl__calloc(gc, 1, len); + ptr = *buf; + + strcpy(ptr, QEMU_HEADER); + ptr += sizeof(QEMU_HEADER); + + memcpy(ptr, &count, sizeof(count)); + ptr += sizeof(count); + + for (i = 0; i < count; i++) { + phys_offset = entries[i]; + start_addr = libxl__xs_read(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%s/start_addr", + domid, phys_offset)); + size = libxl__xs_read(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%s/size", + domid, phys_offset)); + + if (start_addr == NULL || size == NULL || phys_offset == NULL) + return 0; + + val = strtoll(phys_offset, NULL, 16); + memcpy(ptr, &val, sizeof(val)); + ptr += sizeof(val); + val = strtoll(start_addr, NULL, 16); + memcpy(ptr, &val, sizeof(val)); + ptr += sizeof(val); + val = strtoll(size, NULL, 16); + memcpy(ptr, &val, sizeof(val)); + ptr += sizeof(val); + } + + return len; +} + int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) { libxl_ctx *ctx = libxl__gc_owner(gc); - int ret, fd2 = -1, c; - char buf[1024]; + int ret, fd2 = -1, c, len; + char buf[1024], *buf2; const char *filename = libxl__device_model_savefile(gc, domid); struct stat st; uint32_t qemu_state_len; @@ -687,7 +775,8 @@ int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) goto out; } - qemu_state_len = st.st_size; + len = libxl__domain_save_qemu_header(gc, domid, fd, &buf2); + qemu_state_len = st.st_size + len; LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Qemu state is %d bytes\n", qemu_state_len); ret = libxl_write_exactly(ctx, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE), @@ -700,6 +789,11 @@ int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) if (ret) goto out; + ret = libxl_write_exactly(ctx, fd, buf2, len, + "saved-state file", "saved-state qemu header"); + if (ret) + goto out; + fd2 = open(filename, O_RDONLY); if (fd2 < 0) { LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Unable to open qemu save file\n"); diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index 7f7578a..01f866f 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -57,6 +57,7 @@ #define LIBXL_HVM_EXTRA_MEMORY 2048 #define LIBXL_MIN_DOM0_MEM (128*1024) #define QEMU_SIGNATURE "DeviceModelRecord0002" +#define QEMU_HEADER "DeviceModelHeader0001" #define STUBDOM_CONSOLE_LOGGING 0 #define STUBDOM_CONSOLE_SAVE 1 #define STUBDOM_CONSOLE_RESTORE 2 -- 1.7.2.5
Stefano Stabellini
2012-Jan-19 12:13 UTC
[PATCH 1/3] libxc: add a callback to xc_domain_restore
Introduce a callback argument to xc_domain_restore, so that the caller can restore the Qemu state file by itself. If the callback is NULL, libxc will take care of dumping the qemu state file as usual. Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> --- tools/libxc/xc_domain_restore.c | 19 ++++++++++++++----- tools/libxc/xenguest.h | 17 ++++++++++++++--- tools/libxl/libxl_dom.c | 2 +- tools/xcutils/xc_restore.c | 3 ++- 4 files changed, 31 insertions(+), 10 deletions(-) diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c index 14451d1..fdc3483 100644 --- a/tools/libxc/xc_domain_restore.c +++ b/tools/libxc/xc_domain_restore.c @@ -1248,7 +1248,8 @@ static int apply_batch(xc_interface *xch, uint32_t dom, struct restore_ctx *ctx, int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, unsigned int store_evtchn, unsigned long *store_mfn, unsigned int console_evtchn, unsigned long *console_mfn, - unsigned int hvm, unsigned int pae, int superpages) + unsigned int hvm, unsigned int pae, int superpages, + struct restore_callbacks* callbacks) { DECLARE_DOMCTL; int rc = 1, frc, i, j, n, m, pae_extended_cr3 = 0, ext_vcpucontext = 0; @@ -1976,10 +1977,18 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, goto out; finish_hvm: - /* Dump the QEMU state to a state file for QEMU to load */ - if ( dump_qemu(xch, dom, &tailbuf.u.hvm) ) { - PERROR("Error dumping QEMU state to file"); - goto out; + if (callbacks == NULL || callbacks->extract_qemu_savestate == NULL) { + /* Dump the QEMU state to a state file for QEMU to load */ + if ( dump_qemu(xch, dom, &tailbuf.u.hvm) ) { + PERROR("Error dumping QEMU state to file"); + goto out; + } + } else { + if ( callbacks->extract_qemu_savestate(tailbuf.u.hvm.qemubuf, + tailbuf.u.hvm.qemubufsize, callbacks->data) ) { + PERROR("Error calling extract_qemu_savestate"); + goto out; + } } /* These comms pages need to be zeroed at the start of day */ diff --git a/tools/libxc/xenguest.h b/tools/libxc/xenguest.h index 4475ee9..9447412 100644 --- a/tools/libxc/xenguest.h +++ b/tools/libxc/xenguest.h @@ -61,6 +61,15 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter struct save_callbacks* callbacks, int hvm); +/* callbacks provided by xc_domain_restore */ +struct restore_callbacks { + /* callback to extract the qemu save state */ + int (*extract_qemu_savestate)(uint8_t *buf, uint32_t size, void* data); + + /* to be provided as the last argument to each callback function */ + void* data; +}; + /** * This function will restore a saved domain. * @@ -72,15 +81,17 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter * @parm hvm non-zero if this is a HVM restore * @parm pae non-zero if this HVM domain has PAE support enabled * @parm superpages non-zero to allocate guest memory with superpages + * @parm callbacks non-NULL to receive a callback to extract the qemu state * @return 0 on success, -1 on failure */ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, unsigned int store_evtchn, unsigned long *store_mfn, unsigned int console_evtchn, unsigned long *console_mfn, - unsigned int hvm, unsigned int pae, int superpages); + unsigned int hvm, unsigned int pae, int superpages, + struct restore_callbacks *callbacks); /** - * xc_domain_restore writes a file to disk that contains the device - * model saved state. + * If callbacks is NULL, xc_domain_restore writes a file to disk that + * contains the device model saved state. * The pathname of this file is XC_DEVICE_MODEL_RESTORE_FILE; The domid * of the new domain is automatically appended to the filename, * separated by a ".". diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c index c898d89..fd2b051 100644 --- a/tools/libxl/libxl_dom.c +++ b/tools/libxl/libxl_dom.c @@ -373,7 +373,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, rc = xc_domain_restore(ctx->xch, fd, domid, state->store_port, &state->store_mfn, state->console_port, &state->console_mfn, - hvm, pae, superpages); + hvm, pae, superpages, NULL); if ( rc ) { LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain"); return ERROR_FAIL; diff --git a/tools/xcutils/xc_restore.c b/tools/xcutils/xc_restore.c index 985cbec..eb1d31c 100644 --- a/tools/xcutils/xc_restore.c +++ b/tools/xcutils/xc_restore.c @@ -46,7 +46,8 @@ main(int argc, char **argv) superpages = !!hvm; ret = xc_domain_restore(xch, io_fd, domid, store_evtchn, &store_mfn, - console_evtchn, &console_mfn, hvm, pae, superpages); + console_evtchn, &console_mfn, hvm, pae, superpages, + NULL); if ( ret == 0 ) { -- 1.7.2.5
Stefano Stabellini
2012-Jan-19 12:13 UTC
[PATCH 2/3] libxl: extract the qemu state file from the save image
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> --- tools/libxl/libxl_dm.c | 5 +++++ tools/libxl/libxl_dom.c | 42 +++++++++++++++++++++++++++++++++++++++++- tools/libxl/libxl_internal.h | 1 + 3 files changed, 47 insertions(+), 1 deletions(-) diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c index 97d91b4..3f19150 100644 --- a/tools/libxl/libxl_dm.c +++ b/tools/libxl/libxl_dm.c @@ -41,6 +41,11 @@ const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid) return libxl__sprintf(gc, "/var/lib/xen/qemu-save.%d", domid); } +const char *libxl__device_model_restorefile(libxl__gc *gc, uint32_t domid) +{ + return libxl__sprintf(gc, "/var/lib/xen/qemu-resume.%d", domid); +} + const char *libxl__domain_device_model(libxl__gc *gc, libxl_device_model_info *info) { diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c index fd2b051..f33e572 100644 --- a/tools/libxl/libxl_dom.c +++ b/tools/libxl/libxl_dom.c @@ -347,6 +347,40 @@ out: return rc; } +struct restore_callbacks_arg { + libxl__gc *gc; + uint32_t domid; +}; + +static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, + void *data) +{ + struct restore_callbacks_arg *arg = data; + libxl__gc *gc = (libxl__gc *)arg->gc; + libxl_ctx *ctx = libxl__gc_owner(gc); + uint32_t domid = arg->domid; + int fd2 = -1, ret = 0; + const char *filename; + + filename = libxl__device_model_restorefile(gc, domid); + fd2 = open(filename, O_WRONLY|O_CREAT); + if (fd2 < 0) { + ret = fd2; + goto out; + } + + ret = libxl_write_exactly( + ctx, fd2, buf, size, "saved-state file", "qemu state"); + if (ret) + goto out; + ret = 0; + +out: + if (fd2 > 0) + close(fd2); + return ret; +} + int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, libxl_domain_build_info *info, libxl__domain_build_state *state, @@ -356,11 +390,17 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, /* read signature */ int rc; int hvm, pae, superpages; + struct restore_callbacks callbacks; + struct restore_callbacks_arg arg; switch (info->type) { case LIBXL_DOMAIN_TYPE_HVM: hvm = 1; superpages = 1; pae = info->u.hvm.pae; + arg.domid = domid; + arg.gc = gc; + callbacks.extract_qemu_savestate = libxl__domain_restore_device_model; + callbacks.data = &arg; break; case LIBXL_DOMAIN_TYPE_PV: hvm = 0; @@ -373,7 +413,7 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid, rc = xc_domain_restore(ctx->xch, fd, domid, state->store_port, &state->store_mfn, state->console_port, &state->console_mfn, - hvm, pae, superpages, NULL); + hvm, pae, superpages, &callbacks); if ( rc ) { LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "restoring domain"); return ERROR_FAIL; diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index fa7fb16..7f7578a 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -247,6 +247,7 @@ _hidden int libxl__domain_suspend_common(libxl__gc *gc, uint32_t domid, int fd, libxl_domain_type type, int live, int debug); _hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid); +_hidden const char *libxl__device_model_restorefile(libxl__gc *gc, uint32_t domid); _hidden int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd); _hidden void libxl__userdata_destroyall(libxl__gc *gc, uint32_t domid); -- 1.7.2.5
Introduce a new QEMU_HEADER stored in the Qemu chunk right after the QEMU_SIGNATURE and record length, before the Qemu state, to preserve the physmap informations written by Qemu on xenstore. Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> --- tools/libxl/libxl_dom.c | 104 ++++++++++++++++++++++++++++++++++++++++-- tools/libxl/libxl_internal.h | 1 + 2 files changed, 100 insertions(+), 5 deletions(-) diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c index f33e572..0f3d0c3 100644 --- a/tools/libxl/libxl_dom.c +++ b/tools/libxl/libxl_dom.c @@ -359,9 +359,47 @@ static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, libxl__gc *gc = (libxl__gc *)arg->gc; libxl_ctx *ctx = libxl__gc_owner(gc); uint32_t domid = arg->domid; - int fd2 = -1, ret = 0; + int fd2 = -1, ret = 0, i = 0; const char *filename; + uint8_t *ptr = buf; + uint32_t count = 0; + uint64_t phys_offset_v = 0, start_addr_v = 0, size_v = 0; + uint32_t header_size = 0; + + if (strncmp((char *)ptr, QEMU_HEADER, size)) + goto no_header; + + ptr += sizeof(QEMU_HEADER); + memcpy(&count, ptr, sizeof(count)); + ptr += sizeof(count); + header_size = sizeof(QEMU_HEADER) + sizeof(count) + count * sizeof(uint64_t) * 3; + size -= header_size; + + for (i = 0; i < count; i++) { + memcpy(&phys_offset_v, ptr, sizeof(uint64_t)); + ptr += sizeof(uint64_t); + memcpy(&start_addr_v, ptr, sizeof(uint64_t)); + ptr += sizeof(uint64_t); + memcpy(&size_v, ptr, sizeof(uint64_t)); + ptr += sizeof(uint64_t); + + ret = libxl__xs_write(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%"PRIx64"/start_addr", + domid, phys_offset_v), "%"PRIx64, start_addr_v); + if (ret) { + ptr = buf + header_size; + break; + } + ret = libxl__xs_write(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%"PRIx64"/size", + domid, phys_offset_v), "%"PRIx64, size_v); + if (ret) { + ptr = buf + header_size; + break; + } + } +no_header: filename = libxl__device_model_restorefile(gc, domid); fd2 = open(filename, O_WRONLY|O_CREAT); if (fd2 < 0) { @@ -370,7 +408,7 @@ static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, } ret = libxl_write_exactly( - ctx, fd2, buf, size, "saved-state file", "qemu state"); + ctx, fd2, ptr, size, "saved-state file", "qemu state"); if (ret) goto out; ret = 0; @@ -642,11 +680,61 @@ out: return rc; } +static int libxl__domain_save_qemu_header(libxl__gc *gc, uint32_t domid, + int fd, char **buf) +{ + char *start_addr = NULL, *size = NULL, *phys_offset = NULL; + int len = 0, i = 0; + unsigned int num = 0; + uint32_t count = 0; + char *ptr = NULL, **entries = NULL; + uint64_t val = 0; + + entries = libxl__xs_directory(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap", domid), &num); + count = num; + + len = sizeof(QEMU_HEADER) + sizeof(count) + count * (sizeof(val) * 3); + *buf = libxl__calloc(gc, 1, len); + ptr = *buf; + + strcpy(ptr, QEMU_HEADER); + ptr += sizeof(QEMU_HEADER); + + memcpy(ptr, &count, sizeof(count)); + ptr += sizeof(count); + + for (i = 0; i < count; i++) { + phys_offset = entries[i]; + start_addr = libxl__xs_read(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%s/start_addr", + domid, phys_offset)); + size = libxl__xs_read(gc, 0, libxl__sprintf(gc, + "/local/domain/0/device-model/%d/physmap/%s/size", + domid, phys_offset)); + + if (start_addr == NULL || size == NULL || phys_offset == NULL) + return 0; + + val = strtoll(phys_offset, NULL, 16); + memcpy(ptr, &val, sizeof(val)); + ptr += sizeof(val); + val = strtoll(start_addr, NULL, 16); + memcpy(ptr, &val, sizeof(val)); + ptr += sizeof(val); + val = strtoll(size, NULL, 16); + memcpy(ptr, &val, sizeof(val)); + ptr += sizeof(val); + } + + return len; +} + int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) { libxl_ctx *ctx = libxl__gc_owner(gc); - int ret, fd2 = -1, c; - char buf[1024]; + int ret, fd2 = -1, c, len; + char buf[1024], *buf2; const char *filename = libxl__device_model_savefile(gc, domid); struct stat st; uint32_t qemu_state_len; @@ -687,7 +775,8 @@ int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) goto out; } - qemu_state_len = st.st_size; + len = libxl__domain_save_qemu_header(gc, domid, fd, &buf2); + qemu_state_len = st.st_size + len; LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Qemu state is %d bytes\n", qemu_state_len); ret = libxl_write_exactly(ctx, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE), @@ -700,6 +789,11 @@ int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) if (ret) goto out; + ret = libxl_write_exactly(ctx, fd, buf2, len, + "saved-state file", "saved-state qemu header"); + if (ret) + goto out; + fd2 = open(filename, O_RDONLY); if (fd2 < 0) { LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Unable to open qemu save file\n"); diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index 7f7578a..01f866f 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -57,6 +57,7 @@ #define LIBXL_HVM_EXTRA_MEMORY 2048 #define LIBXL_MIN_DOM0_MEM (128*1024) #define QEMU_SIGNATURE "DeviceModelRecord0002" +#define QEMU_HEADER "DeviceModelHeader0001" #define STUBDOM_CONSOLE_LOGGING 0 #define STUBDOM_CONSOLE_SAVE 1 #define STUBDOM_CONSOLE_RESTORE 2 -- 1.7.2.5
On Thu, 2012-01-19 at 12:13 +0000, Stefano Stabellini wrote:> Introduce a new QEMU_HEADER stored in the Qemu chunk right after the > QEMU_SIGNATURE and record length, before the Qemu state, to preserve the > physmap informations written by Qemu on xenstore.I''m still of the opinion that this should be a new XC_SAVE_ID_* (possibly from a new range reserved for toolstakc use) and saved/restored via a separate callback from libxc. The mess that is the tail of the save/restore protocol is pure historical and for backwards compat, we should not be making it any worse. An XC_SAVE_ID is exactly equivalent to adding something to the tail but uses the extensibility mechanism which we have now built into the protocol. Ian.> > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > --- > tools/libxl/libxl_dom.c | 104 ++++++++++++++++++++++++++++++++++++++++-- > tools/libxl/libxl_internal.h | 1 + > 2 files changed, 100 insertions(+), 5 deletions(-) > > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index f33e572..0f3d0c3 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -359,9 +359,47 @@ static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, > libxl__gc *gc = (libxl__gc *)arg->gc; > libxl_ctx *ctx = libxl__gc_owner(gc); > uint32_t domid = arg->domid; > - int fd2 = -1, ret = 0; > + int fd2 = -1, ret = 0, i = 0; > const char *filename; > + uint8_t *ptr = buf; > + uint32_t count = 0; > + uint64_t phys_offset_v = 0, start_addr_v = 0, size_v = 0; > + uint32_t header_size = 0; > + > + if (strncmp((char *)ptr, QEMU_HEADER, size)) > + goto no_header; > + > + ptr += sizeof(QEMU_HEADER); > + memcpy(&count, ptr, sizeof(count)); > + ptr += sizeof(count); > + header_size = sizeof(QEMU_HEADER) + sizeof(count) + count * sizeof(uint64_t) * 3; > + size -= header_size; > + > + for (i = 0; i < count; i++) { > + memcpy(&phys_offset_v, ptr, sizeof(uint64_t)); > + ptr += sizeof(uint64_t); > + memcpy(&start_addr_v, ptr, sizeof(uint64_t)); > + ptr += sizeof(uint64_t); > + memcpy(&size_v, ptr, sizeof(uint64_t)); > + ptr += sizeof(uint64_t); > + > + ret = libxl__xs_write(gc, 0, libxl__sprintf(gc, > + "/local/domain/0/device-model/%d/physmap/%"PRIx64"/start_addr", > + domid, phys_offset_v), "%"PRIx64, start_addr_v); > + if (ret) { > + ptr = buf + header_size; > + break; > + } > + ret = libxl__xs_write(gc, 0, libxl__sprintf(gc, > + "/local/domain/0/device-model/%d/physmap/%"PRIx64"/size", > + domid, phys_offset_v), "%"PRIx64, size_v); > + if (ret) { > + ptr = buf + header_size; > + break; > + } > + } > > +no_header: > filename = libxl__device_model_restorefile(gc, domid); > fd2 = open(filename, O_WRONLY|O_CREAT); > if (fd2 < 0) { > @@ -370,7 +408,7 @@ static int libxl__domain_restore_device_model(uint8_t *buf, uint32_t size, > } > > ret = libxl_write_exactly( > - ctx, fd2, buf, size, "saved-state file", "qemu state"); > + ctx, fd2, ptr, size, "saved-state file", "qemu state"); > if (ret) > goto out; > ret = 0; > @@ -642,11 +680,61 @@ out: > return rc; > } > > +static int libxl__domain_save_qemu_header(libxl__gc *gc, uint32_t domid, > + int fd, char **buf) > +{ > + char *start_addr = NULL, *size = NULL, *phys_offset = NULL; > + int len = 0, i = 0; > + unsigned int num = 0; > + uint32_t count = 0; > + char *ptr = NULL, **entries = NULL; > + uint64_t val = 0; > + > + entries = libxl__xs_directory(gc, 0, libxl__sprintf(gc, > + "/local/domain/0/device-model/%d/physmap", domid), &num); > + count = num; > + > + len = sizeof(QEMU_HEADER) + sizeof(count) + count * (sizeof(val) * 3); > + *buf = libxl__calloc(gc, 1, len); > + ptr = *buf; > + > + strcpy(ptr, QEMU_HEADER); > + ptr += sizeof(QEMU_HEADER); > + > + memcpy(ptr, &count, sizeof(count)); > + ptr += sizeof(count); > + > + for (i = 0; i < count; i++) { > + phys_offset = entries[i]; > + start_addr = libxl__xs_read(gc, 0, libxl__sprintf(gc, > + "/local/domain/0/device-model/%d/physmap/%s/start_addr", > + domid, phys_offset)); > + size = libxl__xs_read(gc, 0, libxl__sprintf(gc, > + "/local/domain/0/device-model/%d/physmap/%s/size", > + domid, phys_offset)); > + > + if (start_addr == NULL || size == NULL || phys_offset == NULL) > + return 0; > + > + val = strtoll(phys_offset, NULL, 16); > + memcpy(ptr, &val, sizeof(val)); > + ptr += sizeof(val); > + val = strtoll(start_addr, NULL, 16); > + memcpy(ptr, &val, sizeof(val)); > + ptr += sizeof(val); > + val = strtoll(size, NULL, 16); > + memcpy(ptr, &val, sizeof(val)); > + ptr += sizeof(val); > + } > + > + return len; > +} > + > int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) > { > libxl_ctx *ctx = libxl__gc_owner(gc); > - int ret, fd2 = -1, c; > - char buf[1024]; > + int ret, fd2 = -1, c, len; > + char buf[1024], *buf2; > const char *filename = libxl__device_model_savefile(gc, domid); > struct stat st; > uint32_t qemu_state_len; > @@ -687,7 +775,8 @@ int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) > goto out; > } > > - qemu_state_len = st.st_size; > + len = libxl__domain_save_qemu_header(gc, domid, fd, &buf2); > + qemu_state_len = st.st_size + len; > LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Qemu state is %d bytes\n", qemu_state_len); > > ret = libxl_write_exactly(ctx, fd, QEMU_SIGNATURE, strlen(QEMU_SIGNATURE), > @@ -700,6 +789,11 @@ int libxl__domain_save_device_model(libxl__gc *gc, uint32_t domid, int fd) > if (ret) > goto out; > > + ret = libxl_write_exactly(ctx, fd, buf2, len, > + "saved-state file", "saved-state qemu header"); > + if (ret) > + goto out; > + > fd2 = open(filename, O_RDONLY); > if (fd2 < 0) { > LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Unable to open qemu save file\n"); > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index 7f7578a..01f866f 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -57,6 +57,7 @@ > #define LIBXL_HVM_EXTRA_MEMORY 2048 > #define LIBXL_MIN_DOM0_MEM (128*1024) > #define QEMU_SIGNATURE "DeviceModelRecord0002" > +#define QEMU_HEADER "DeviceModelHeader0001" > #define STUBDOM_CONSOLE_LOGGING 0 > #define STUBDOM_CONSOLE_SAVE 1 > #define STUBDOM_CONSOLE_RESTORE 2
Ian Campbell
2012-Jan-19 12:25 UTC
Re: [PATCH 0/3] libxl: save additional info to the qemu chunk
On Thu, 2012-01-19 at 12:12 +0000, Stefano Stabellini wrote:> Stefano Stabellini (3): > libxc: add a callback to xc_domain_restore > libxl: extract the qemu state file from the save image > libxl: introduce QEMU_HEADERTwo sets of these followed. I''ve assumed they were identical.
On Thu, 19 Jan 2012, Ian Campbell wrote:> On Thu, 2012-01-19 at 12:13 +0000, Stefano Stabellini wrote: > > Introduce a new QEMU_HEADER stored in the Qemu chunk right after the > > QEMU_SIGNATURE and record length, before the Qemu state, to preserve the > > physmap informations written by Qemu on xenstore. > > I''m still of the opinion that this should be a new XC_SAVE_ID_* > (possibly from a new range reserved for toolstakc use) and > saved/restored via a separate callback from libxc. > > The mess that is the tail of the save/restore protocol is pure > historical and for backwards compat, we should not be making it any > worse. An XC_SAVE_ID is exactly equivalent to adding something to the > tail but uses the extensibility mechanism which we have now built into > the protocol.OK, I''ll give it a look. I wasn''t sure what was legacy and what was not...
Stefano Stabellini
2012-Jan-19 14:00 UTC
Re: [PATCH 0/3] libxl: save additional info to the qemu chunk
On Thu, 19 Jan 2012, Ian Campbell wrote:> On Thu, 2012-01-19 at 12:12 +0000, Stefano Stabellini wrote: > > > Stefano Stabellini (3): > > libxc: add a callback to xc_domain_restore > > libxl: extract the qemu state file from the save image > > libxl: introduce QEMU_HEADER > > Two sets of these followed. I''ve assumed they were identical.Uhm.. I only executed git send-email once, I am not sure what is going on..
Ian Campbell
2012-Jan-19 14:01 UTC
Re: [PATCH 0/3] libxl: save additional info to the qemu chunk
On Thu, 2012-01-19 at 14:00 +0000, Stefano Stabellini wrote:> On Thu, 19 Jan 2012, Ian Campbell wrote: > > On Thu, 2012-01-19 at 12:12 +0000, Stefano Stabellini wrote: > > > > > Stefano Stabellini (3): > > > libxc: add a callback to xc_domain_restore > > > libxl: extract the qemu state file from the save image > > > libxl: introduce QEMU_HEADER > > > > Two sets of these followed. I''ve assumed they were identical. > > Uhm.. I only executed git send-email once, I am not sure what is going > on..Weird, the message IDs go from 1-6. Do you have two copies of the patches in your dir + did you use a wild card on your send-email maybe? Ian.
Stefano Stabellini
2012-Jan-19 14:06 UTC
Re: [PATCH 0/3] libxl: save additional info to the qemu chunk
On Thu, 19 Jan 2012, Ian Campbell wrote:> On Thu, 2012-01-19 at 14:00 +0000, Stefano Stabellini wrote: > > On Thu, 19 Jan 2012, Ian Campbell wrote: > > > On Thu, 2012-01-19 at 12:12 +0000, Stefano Stabellini wrote: > > > > > > > Stefano Stabellini (3): > > > > libxc: add a callback to xc_domain_restore > > > > libxl: extract the qemu state file from the save image > > > > libxl: introduce QEMU_HEADER > > > > > > Two sets of these followed. I''ve assumed they were identical. > > > > Uhm.. I only executed git send-email once, I am not sure what is going > > on.. > > Weird, the message IDs go from 1-6. > > Do you have two copies of the patches in your dir + did you use a wild > card on your send-email maybe?I think I passed the wild card to git send-email twice, my fault :/
Ian Campbell
2012-Jan-19 14:09 UTC
Re: [PATCH 0/3] libxl: save additional info to the qemu chunk
On Thu, 2012-01-19 at 14:06 +0000, Stefano Stabellini wrote:> On Thu, 19 Jan 2012, Ian Campbell wrote: > > On Thu, 2012-01-19 at 14:00 +0000, Stefano Stabellini wrote: > > > On Thu, 19 Jan 2012, Ian Campbell wrote: > > > > On Thu, 2012-01-19 at 12:12 +0000, Stefano Stabellini wrote: > > > > > > > > > Stefano Stabellini (3): > > > > > libxc: add a callback to xc_domain_restore > > > > > libxl: extract the qemu state file from the save image > > > > > libxl: introduce QEMU_HEADER > > > > > > > > Two sets of these followed. I''ve assumed they were identical. > > > > > > Uhm.. I only executed git send-email once, I am not sure what is going > > > on.. > > > > Weird, the message IDs go from 1-6. > > > > Do you have two copies of the patches in your dir + did you use a wild > > card on your send-email maybe? > > I think I passed the wild card to git send-email twice, my fault :/Easily done. FWIW I usually do less $(git format-patch RANGE..RANGE) git send-email --to etc $(git format-patch RANGE..RANGE) etc to avoid this sort of issue. Ian.