Hi all,
I''m interested in developing live migration in xen arm and possibly
the contribution to the community and I hope this patch series could be a start.
For this matter, I have following questions:
(1) Is it OK to keep using the keyword "hvm"? Or, is it better to use
pvh?
(2) After some overview of source code, I think the required parts
for save/restore are the following:
- xen-store info
- shared info page
- memory contents (no need for p2m table)
- cpu/vcpu
- gic/vgic
- drivers
I think there are still important parts that I''m missing.
I appreciate if you could give some advice :)
(3) Regarding split drivers, come to think of it, we have to store
both side (front/back) states, in-flight event channels, IRQs, etc.
And those look like quite a work (although evtchn is migrated within vcpu)
I appreciate if you guys could share any hints from the experience of
migrating split drivers in x86.
Lastly I would like to note that the following patch series is just the
concept work for reviewing my idea and they are quite preliminary.
Jaeyong Yoo (4):
Create new directory for stroing hvm-related files in ARM.
Implement arch_hvm_save and arch_hvm_load functions
Implement save and restore for gic (template impl)
Implement XEN_DOMCTL_gethvmcontext part of arch_do_domctl
xen/arch/arm/Makefile | 2 +-
xen/arch/arm/domctl.c | 58 +++++++++++++++-
xen/arch/arm/hvm.c | 67 ------------------
xen/arch/arm/hvm/Makefile | 2 +
xen/arch/arm/hvm/hvm.c | 118 ++++++++++++++++++++++++++++++++
xen/arch/arm/hvm/save.c | 69 +++++++++++++++++++
xen/common/Makefile | 2 +
xen/include/asm-arm/hvm/support.h | 29 ++++++++
xen/include/public/arch-arm/hvm/save.h | 36 ++++++++++
9 files changed, 314 insertions(+), 69 deletions(-)
delete mode 100644 xen/arch/arm/hvm.c
create mode 100644 xen/arch/arm/hvm/Makefile
create mode 100644 xen/arch/arm/hvm/hvm.c
create mode 100644 xen/arch/arm/hvm/save.c
create mode 100644 xen/include/asm-arm/hvm/support.h
--
1.7.9.5
Jaeyong Yoo
2013-Jun-05 05:46 UTC
[PATCH RFC 1/4] Create new directory for stroing hvm-related files in ARM.
Signed-off-by: Jaeyong Yoo <jaeyong.yoo@samsung.com>
---
xen/arch/arm/Makefile | 2 +-
xen/arch/arm/hvm/Makefile | 1 +
xen/arch/arm/{ => hvm}/hvm.c | 0
3 files changed, 2 insertions(+), 1 deletion(-)
create mode 100644 xen/arch/arm/hvm/Makefile
rename xen/arch/arm/{ => hvm}/hvm.c (100%)
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 87fabe1..0d43539 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -1,6 +1,7 @@
subdir-$(arm32) += arm32
subdir-$(arm64) += arm64
subdir-y += platforms
+subdir-y += hvm
obj-$(EARLY_PRINTK) += early_printk.o
obj-y += cpu.o
@@ -28,7 +29,6 @@ obj-y += traps.o
obj-y += vgic.o
obj-y += vtimer.o
obj-y += vpl011.o
-obj-y += hvm.o
obj-y += device.o
#obj-bin-y += ....o
diff --git a/xen/arch/arm/hvm/Makefile b/xen/arch/arm/hvm/Makefile
new file mode 100644
index 0000000..6ee4054
--- /dev/null
+++ b/xen/arch/arm/hvm/Makefile
@@ -0,0 +1 @@
+obj-y += hvm.o
diff --git a/xen/arch/arm/hvm.c b/xen/arch/arm/hvm/hvm.c
similarity index 100%
rename from xen/arch/arm/hvm.c
rename to xen/arch/arm/hvm/hvm.c
--
1.7.9.5
Jaeyong Yoo
2013-Jun-05 05:46 UTC
[PATCH RFC 2/4] Implement arch_hvm_save and arch_hvm_load functions
Signed-off-by: Jaeyong Yoo <jaeyong.yoo@samsung.com>
---
xen/arch/arm/hvm/Makefile | 1 +
xen/arch/arm/hvm/save.c | 69 ++++++++++++++++++++++++++++++++
xen/include/public/arch-arm/hvm/save.h | 16 ++++++++
3 files changed, 86 insertions(+)
create mode 100644 xen/arch/arm/hvm/save.c
diff --git a/xen/arch/arm/hvm/Makefile b/xen/arch/arm/hvm/Makefile
index 6ee4054..ad38e09 100644
--- a/xen/arch/arm/hvm/Makefile
+++ b/xen/arch/arm/hvm/Makefile
@@ -1 +1,2 @@
obj-y += hvm.o
+obj-y += save.o
diff --git a/xen/arch/arm/hvm/save.c b/xen/arch/arm/hvm/save.c
new file mode 100644
index 0000000..b92f593
--- /dev/null
+++ b/xen/arch/arm/hvm/save.c
@@ -0,0 +1,69 @@
+/*
+ * hvm/save.c: Save and restore HVM guest''s emulated hardware state
for ARM.
+ *
+ * Copyright (c) 2013, Samsung Electronics.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ */
+
+#include <xen/types.h>
+#include <xen/hvm/save.h>
+#include <public/arch-arm/hvm/save.h>
+#include <asm/processor.h>
+#include <xen/sched.h>
+
+void arch_hvm_save(struct domain *d, struct hvm_save_header *hdr)
+{
+ hdr->cpuid = READ_SYSREG32(MIDR_EL1);
+}
+
+int arch_hvm_load(struct domain *d, struct hvm_save_header *hdr)
+{
+ uint32_t cpuid;
+
+ if ( hdr->magic != HVM_FILE_MAGIC )
+ {
+ printk(XENLOG_G_ERR "HVM%d restore: bad magic number
%#"PRIx32"\n",
+ d->domain_id, hdr->magic);
+ return -1;
+ }
+
+ if ( hdr->version != HVM_FILE_VERSION )
+ {
+ printk(XENLOG_G_ERR "HVM%d restore: unsupported version
%u\n",
+ d->domain_id, hdr->version);
+ return -1;
+ }
+
+ cpuid = READ_SYSREG32(MIDR_EL1);
+ if ( hdr->cpuid != cpuid )
+ {
+ printk(XENLOG_G_INFO "HVM%d restore: VM saved on one CPU "
+ "(%#"PRIx32") and restored on another
(%#"PRIx32").\n",
+ d->domain_id, hdr->cpuid, cpuid);
+ return -1;
+ }
+
+ return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/arch-arm/hvm/save.h
b/xen/include/public/arch-arm/hvm/save.h
index 75b8e65..570809e 100644
--- a/xen/include/public/arch-arm/hvm/save.h
+++ b/xen/include/public/arch-arm/hvm/save.h
@@ -26,6 +26,22 @@
#ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__
#define __XEN_PUBLIC_HVM_SAVE_ARM_H__
+/*
+ * Save/restore header: general info about the save file.
+ */
+
+#define HVM_FILE_MAGIC 0x92385520
+#define HVM_FILE_VERSION 0x00000001
+
+struct hvm_save_header {
+ uint32_t magic; /* Must be HVM_FILE_MAGIC */
+ uint32_t version; /* File format version */
+ uint64_t changeset; /* Version of Xen that saved this file */
+ uint32_t cpuid; /* CPUID[0x01][%eax] on the saving machine */
+};
+
+DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct hvm_save_header);
+
#endif
/*
--
1.7.9.5
Jaeyong Yoo
2013-Jun-05 05:46 UTC
[PATCH RFC 3/4] Implement save and restore for gic (template impl)
Signed-off-by: Jaeyong Yoo <jaeyong.yoo@samsung.com>
---
xen/arch/arm/hvm/hvm.c | 51 ++++++++++++++++++++++++++++++++
xen/common/Makefile | 2 ++
xen/include/asm-arm/hvm/support.h | 29 ++++++++++++++++++
xen/include/public/arch-arm/hvm/save.h | 20 +++++++++++++
4 files changed, 102 insertions(+)
create mode 100644 xen/include/asm-arm/hvm/support.h
diff --git a/xen/arch/arm/hvm/hvm.c b/xen/arch/arm/hvm/hvm.c
index 471c4cd..45cc4fd 100644
--- a/xen/arch/arm/hvm/hvm.c
+++ b/xen/arch/arm/hvm/hvm.c
@@ -7,6 +7,7 @@
#include <xsm/xsm.h>
+#include <xen/hvm/save.h>
#include <public/xen.h>
#include <public/hvm/params.h>
#include <public/hvm/hvm_op.h>
@@ -65,3 +66,53 @@ long do_hvm_op(unsigned long op, XEN_GUEST_HANDLE_PARAM(void)
arg)
return rc;
}
+
+static int gic_save(struct domain *d, hvm_domain_context_t *h)
+{
+ struct hvm_hw_gic ctxt;
+ struct vcpu *v;
+
+ /* Save the state of GICs */
+ for_each_vcpu( d, v )
+ {
+ ctxt.gic_hcr = v->arch.gic_hcr;
+ ctxt.gic_vmcr = v->arch.gic_vmcr;
+ ctxt.gic_apr = v->arch.gic_apr;
+ memcpy( ctxt.gic_lr, v->arch.gic_lr, sizeof(v->arch.gic_lr) );
+ ctxt.event_mask = v->arch.event_mask;
+ ctxt.lr_mask = v->arch.lr_mask;
+ if ( hvm_save_entry(GIC, v->vcpu_id, h, &ctxt) != 0 )
+ return 1;
+ }
+ return 0;
+}
+
+static int gic_load(struct domain *d, hvm_domain_context_t *h)
+{
+ int vcpuid;
+ struct hvm_hw_gic ctxt;
+ struct vcpu *v;
+
+ /* Which vcpu is this? */
+ vcpuid = hvm_load_instance(h);
+ if ( vcpuid >= d->max_vcpus || (v = d->vcpu[vcpuid]) == NULL )
+ {
+ dprintk(XENLOG_G_ERR, "HVM restore: dom%u has no vcpu%u\n",
+ d->domain_id, vcpuid);
+ return -EINVAL;
+ }
+
+ if ( hvm_load_entry(GIC, h, &ctxt) != 0 )
+ return -EINVAL;
+
+ v->arch.gic_hcr = ctxt.gic_hcr;
+ v->arch.gic_vmcr = ctxt.gic_vmcr;
+ v->arch.gic_apr = ctxt.gic_apr;
+ memcpy( v->arch.gic_lr, ctxt.gic_lr, sizeof(v->arch.gic_lr) );
+ v->arch.event_mask = ctxt.event_mask;
+ v->arch.lr_mask = ctxt.lr_mask;
+
+ return 0;
+}
+
+HVM_REGISTER_SAVE_RESTORE(GIC, gic_save, gic_load, 1, HVMSR_PER_VCPU);
diff --git a/xen/common/Makefile b/xen/common/Makefile
index 0dc2050..956cf29 100644
--- a/xen/common/Makefile
+++ b/xen/common/Makefile
@@ -60,6 +60,8 @@ subdir-$(CONFIG_COMPAT) += compat
subdir-$(x86_64) += hvm
+subdir-$(CONFIG_ARM) += hvm
+
subdir-$(coverage) += gcov
subdir-y += libelf
diff --git a/xen/include/asm-arm/hvm/support.h
b/xen/include/asm-arm/hvm/support.h
new file mode 100644
index 0000000..33390b0
--- /dev/null
+++ b/xen/include/asm-arm/hvm/support.h
@@ -0,0 +1,29 @@
+/*
+ * support.h: HVM support routines used by ARMv7 VE.
+ *
+ * Copyright (c) 2013, Samsung Electronics
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ */
+
+#ifndef __ASM_ARM_HVM_SUPPORT_H__
+#define __ASM_ARM_HVM_SUPPORT_H__
+
+#include <xen/types.h>
+#include <public/hvm/ioreq.h>
+#include <xen/sched.h>
+#include <xen/hvm/save.h>
+#include <asm/processor.h>
+
+#endif /* __ASM_ARM_HVM_SUPPORT_H__ */
diff --git a/xen/include/public/arch-arm/hvm/save.h
b/xen/include/public/arch-arm/hvm/save.h
index 570809e..bbd9dba 100644
--- a/xen/include/public/arch-arm/hvm/save.h
+++ b/xen/include/public/arch-arm/hvm/save.h
@@ -42,6 +42,26 @@ struct hvm_save_header {
DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct hvm_save_header);
+
+/*
+ * GIC
+ */
+struct hvm_hw_gic {
+ uint32_t gic_hcr;
+ uint32_t gic_vmcr;
+ uint32_t gic_apr;
+ uint32_t gic_lr[64];
+ uint64_t event_mask;
+ uint64_t lr_mask;
+};
+
+DECLARE_HVM_SAVE_TYPE(GIC, 3, struct hvm_hw_gic); /* leave 2 for CPU */
+
+/*
+ * Largest type-code in use
+ */
+#define HVM_SAVE_CODE_MAX 19
+
#endif
/*
--
1.7.9.5
Jaeyong Yoo
2013-Jun-05 05:46 UTC
[PATCH RFC 4/4] Implement XEN_DOMCTL_gethvmcontext part of arch_do_domctl
Signed-off-by: Jaeyong Yoo <jaeyong.yoo@samsung.com>
---
xen/arch/arm/domctl.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 57 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
index 851ee40..a1ff570 100644
--- a/xen/arch/arm/domctl.c
+++ b/xen/arch/arm/domctl.c
@@ -9,12 +9,68 @@
#include <xen/lib.h>
#include <xen/errno.h>
#include <xen/sched.h>
+#include <xen/hvm/save.h>
+#include <xen/guest_access.h>
#include <public/domctl.h>
long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
{
- return -ENOSYS;
+ long ret = 0;
+ bool_t copyback = 0;
+ switch ( domctl->cmd )
+ {
+ case XEN_DOMCTL_gethvmcontext:
+ {
+ struct hvm_domain_context c = { 0 };
+
+ ret = -EINVAL;
+ if ( !is_hvm_domain(d) ) {
+ printk("Domain is not hvm... Just go on...\n");
+ }
+
+ c.size = hvm_save_size(d);
+
+ if ( guest_handle_is_null(domctl->u.hvmcontext.buffer) )
+ {
+ /* Client is querying for the correct buffer size */
+ domctl->u.hvmcontext.size = c.size;
+ ret = 0;
+ goto gethvmcontext_out;
+ }
+
+ /* Check that the client has a big enough buffer */
+ ret = -ENOSPC;
+ if ( domctl->u.hvmcontext.size < c.size )
+ goto gethvmcontext_out;
+
+ /* Allocate our own marshalling buffer */
+ ret = -ENOMEM;
+ if ( (c.data = xmalloc_bytes(c.size)) == NULL )
+ goto gethvmcontext_out;
+
+ domain_pause(d);
+ ret = hvm_save(d, &c);
+ domain_unpause(d);
+
+ domctl->u.hvmcontext.size = c.cur;
+ if ( copy_to_guest(domctl->u.hvmcontext.buffer, c.data, c.size) != 0
)
+ ret = -EFAULT;
+
+ gethvmcontext_out:
+ copyback = 1;
+
+ if ( c.data != NULL )
+ xfree(c.data);
+ }
+ break;
+ default:
+ return -EINVAL;
+ }
+ if ( copyback && __copy_to_guest(u_domctl, domctl, 1) )
+ ret = -EFAULT;
+
+ return ret;
}
void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
--
1.7.9.5
On Wed, 2013-06-05 at 14:46 +0900, Jaeyong Yoo wrote:> Hi all, > I''m interested in developing live migration in xen arm and possibly > the contribution to the community and I hope this patch series could be a start. > > For this matter, I have following questions: > > (1) Is it OK to keep using the keyword "hvm"? Or, is it better to use pvh?My personal preference is to avoid either of them, they are IMHO x86 specific terms. On ARM we (currently) have only "domains", there aren''t (yet) multiple types of domain> (2) After some overview of source code, I think the required parts > for save/restore are the following: > - xen-store infoThis should be recreated on the far side based on the guest configuration, it doesn''t need to be explicitly saved (this ties into your #3 below)> - shared info pageThis is either handled as part of the memory contents or is reinitialised from scratch on the destination. I forget which but in any case it doesn''t need to be explicitly transferred, I don''t think. Perhaps its guest PFN location does need to be saved.> - memory contents (no need for p2m table) > - cpu/vcpu > - gic/vgicYes> - driversNot sure which drivers you mean. Any emulated hardware would need its state pickling, but PV drivers do not (see below).> I think there are still important parts that I''m missing. > I appreciate if you could give some advice :) > > (3) Regarding split drivers, come to think of it, we have to store > both side (front/back) states, in-flight event channels, IRQs, etc. > And those look like quite a work (although evtchn is migrated within vcpu) > I appreciate if you guys could share any hints from the experience of > migrating split drivers in x86.There is no need to save any of this state, the devices are effectively reconnected from scratch on the destination and the frontend devices are responsible for replaying any inflight state on resume. Ian.> Lastly I would like to note that the following patch series is just the > concept work for reviewing my idea and they are quite preliminary. > > > Jaeyong Yoo (4): > Create new directory for stroing hvm-related files in ARM. > Implement arch_hvm_save and arch_hvm_load functions > Implement save and restore for gic (template impl) > Implement XEN_DOMCTL_gethvmcontext part of arch_do_domctl > > xen/arch/arm/Makefile | 2 +- > xen/arch/arm/domctl.c | 58 +++++++++++++++- > xen/arch/arm/hvm.c | 67 ------------------ > xen/arch/arm/hvm/Makefile | 2 + > xen/arch/arm/hvm/hvm.c | 118 ++++++++++++++++++++++++++++++++ > xen/arch/arm/hvm/save.c | 69 +++++++++++++++++++ > xen/common/Makefile | 2 + > xen/include/asm-arm/hvm/support.h | 29 ++++++++ > xen/include/public/arch-arm/hvm/save.h | 36 ++++++++++ > 9 files changed, 314 insertions(+), 69 deletions(-) > delete mode 100644 xen/arch/arm/hvm.c > create mode 100644 xen/arch/arm/hvm/Makefile > create mode 100644 xen/arch/arm/hvm/hvm.c > create mode 100644 xen/arch/arm/hvm/save.c > create mode 100644 xen/include/asm-arm/hvm/support.h >
Ian Campbell
2013-Jun-05 09:17 UTC
Re: [PATCH RFC 2/4] Implement arch_hvm_save and arch_hvm_load functions
> diff --git a/xen/include/public/arch-arm/hvm/save.h b/xen/include/public/arch-arm/hvm/save.h > index 75b8e65..570809e 100644 > --- a/xen/include/public/arch-arm/hvm/save.h > +++ b/xen/include/public/arch-arm/hvm/save.h > @@ -26,6 +26,22 @@ > #ifndef __XEN_PUBLIC_HVM_SAVE_ARM_H__ > #define __XEN_PUBLIC_HVM_SAVE_ARM_H__ > > +/* > + * Save/restore header: general info about the save file. > + */ > + > +#define HVM_FILE_MAGIC 0x92385520 > +#define HVM_FILE_VERSION 0x00000001 > + > +struct hvm_save_header { > + uint32_t magic; /* Must be HVM_FILE_MAGIC */ > + uint32_t version; /* File format version */ > + uint64_t changeset; /* Version of Xen that saved this file */ > + uint32_t cpuid; /* CPUID[0x01][%eax] on the saving machine */The comment here looks x86 to be specific. Need to consider whether MIDR is sufficient for the safety check we have or whether it is appropriate in the header at all, perhaps it should be in a separate cpu features save header?
Ian Campbell
2013-Jun-05 09:21 UTC
Re: [PATCH RFC 3/4] Implement save and restore for gic (template impl)
> @@ -65,3 +66,53 @@ long do_hvm_op(unsigned long op, XEN_GUEST_HANDLE_PARAM(void) arg) > > return rc; > } > + > +static int gic_save(struct domain *d, hvm_domain_context_t *h) > +{ > + struct hvm_hw_gic ctxt; > + struct vcpu *v; > + > + /* Save the state of GICs */ > + for_each_vcpu( d, v ) > + { > + ctxt.gic_hcr = v->arch.gic_hcr; > + ctxt.gic_vmcr = v->arch.gic_vmcr; > + ctxt.gic_apr = v->arch.gic_apr; > + memcpy( ctxt.gic_lr, v->arch.gic_lr, sizeof(v->arch.gic_lr) ); > + ctxt.event_mask = v->arch.event_mask; > + ctxt.lr_mask = v->arch.lr_mask;In general on x86 the policy is that hvm headers should store only architectural state. I think this is a good policy to carry over to ARM. With that in mind I don''t think either event_mask or lr_mask are architectural state but are actually internal state of the vgic "emulation" which can and should be reconstructed when loading the architectural state. Ian
On Wed, 2013-06-05 at 10:13 +0100, Ian Campbell wrote:> On Wed, 2013-06-05 at 14:46 +0900, Jaeyong Yoo wrote: > > Hi all, > > I''m interested in developing live migration in xen arm and possibly > > the contribution to the community and I hope this patch series could be a start. > > > > For this matter, I have following questions: > > > > (1) Is it OK to keep using the keyword "hvm"? Or, is it better to use pvh? > > My personal preference is to avoid either of them, they are IMHO x86 > specific terms. On ARM we (currently) have only "domains", there aren''t > (yet) multiple types of domainBTW, I appreciate that the term HVM at least is part of the common hypercall API which we share with x86 -- we just have to live with that I think.
Ian Campbell
2013-Jun-05 09:24 UTC
Re: [PATCH RFC 4/4] Implement XEN_DOMCTL_gethvmcontext part of arch_do_domctl
On Wed, 2013-06-05 at 14:46 +0900, Jaeyong Yoo wrote:> Signed-off-by: Jaeyong Yoo <jaeyong.yoo@samsung.com> > --- > xen/arch/arm/domctl.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 57 insertions(+), 1 deletion(-) > > diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c > index 851ee40..a1ff570 100644 > --- a/xen/arch/arm/domctl.c > +++ b/xen/arch/arm/domctl.c > @@ -9,12 +9,68 @@ > #include <xen/lib.h> > #include <xen/errno.h> > #include <xen/sched.h> > +#include <xen/hvm/save.h> > +#include <xen/guest_access.h> > #include <public/domctl.h> > > long arch_do_domctl(struct xen_domctl *domctl, struct domain *d, > XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl) > { > - return -ENOSYS; > + long ret = 0; > + bool_t copyback = 0; > + switch ( domctl->cmd ) > + { > + case XEN_DOMCTL_gethvmcontext: > + { > + struct hvm_domain_context c = { 0 }; > + > + ret = -EINVAL; > + if ( !is_hvm_domain(d) ) { > + printk("Domain is not hvm... Just go on...\n"); > + }I don''t think you need this check.> + > + c.size = hvm_save_size(d); > + > + if ( guest_handle_is_null(domctl->u.hvmcontext.buffer) ) > + { > + /* Client is querying for the correct buffer size */ > + domctl->u.hvmcontext.size = c.size; > + ret = 0; > + goto gethvmcontext_out; > + } > + > + /* Check that the client has a big enough buffer */ > + ret = -ENOSPC; > + if ( domctl->u.hvmcontext.size < c.size ) > + goto gethvmcontext_out; > + > + /* Allocate our own marshalling buffer */ > + ret = -ENOMEM; > + if ( (c.data = xmalloc_bytes(c.size)) == NULL ) > + goto gethvmcontext_out; > + > + domain_pause(d); > + ret = hvm_save(d, &c); > + domain_unpause(d); > + > + domctl->u.hvmcontext.size = c.cur; > + if ( copy_to_guest(domctl->u.hvmcontext.buffer, c.data, c.size) != 0 ) > + ret = -EFAULT; > + > + gethvmcontext_out: > + copyback = 1; > + > + if ( c.data != NULL ) > + xfree(c.data); > + } > + break; > + default: > + return -EINVAL; > + } > + if ( copyback && __copy_to_guest(u_domctl, domctl, 1) ) > + ret = -EFAULT; > + > + return ret; > } > > void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
On Wed, 2013-06-05 at 14:46 +0900, Jaeyong Yoo wrote:> Lastly I would like to note that the following patch series is just the > concept work for reviewing my idea and they are quite preliminary.I took a look and made a few comments but as a preliminary framework it looks good, thanks! BTW, we are currently in feature freeze for the 4.3 release and save/restore is clearly 4.4 material I think, but that shouldn''t stop us from working on the feature it just means it won''t be committed right away. Ian.
> > > For this matter, I have following questions: > > > > > > (1) Is it OK to keep using the keyword "hvm"? Or, is it better to use pvh? > > > > My personal preference is to avoid either of them, they are IMHO x86 > > specific terms. On ARM we (currently) have only "domains", there aren''t > > (yet) multiple types of domain > > BTW, I appreciate that the term HVM at least is part of the common > hypercall API which we share with x86 -- we just have to live with that > I think.Got it. Jaeyong
Hello Ian, I got some more questions while implementing live migration in arm.> > (3) Regarding split drivers, come to think of it, we have to store > > both side (front/back) states, in-flight event channels, IRQs, etc. > > And those look like quite a work (although evtchn is migrated within vcpu) > > I appreciate if you guys could share any hints from the experience of > > migrating split drivers in x86. > > There is no need to save any of this state, the devices are effectively > reconnected from scratch on the destination and the frontend devices are > responsible for replaying any inflight state on resume. >I''m trying to find the suspending part of PV driver, but can not locate it. (And, looks like there is no function like suspend in xen-blkfront.c) One guess is by deleting vbd in xenstore, maybe hot-plug-out can suspend. Could you give me some advice? And, about grant table, if we effectively suspend and resume PV driver, I think we don''t need to migrate grant table entires. Is this right? Thanks, Jaeyong
On Thu, 2013-06-13 at 01:15 +0000, Jaeyong Yoo wrote:> Hello Ian, > I got some more questions while implementing live migration in arm. > > > > (3) Regarding split drivers, come to think of it, we have to store > > > both side (front/back) states, in-flight event channels, IRQs, etc. > > > And those look like quite a work (although evtchn is migrated within vcpu) > > > I appreciate if you guys could share any hints from the experience of > > > migrating split drivers in x86. > > > > There is no need to save any of this state, the devices are effectively > > reconnected from scratch on the destination and the frontend devices are > > responsible for replaying any inflight state on resume. > > > > I''m trying to find the suspending part of PV driver, but can not locate it. > (And, looks like there is no function like suspend in xen-blkfront.c) > One guess is by deleting vbd in xenstore, maybe hot-plug-out can suspend. > Could you give me some advice?IIRC this is all handled by the block driver on resume and no work is required on suspend. Essentially it simply replays the ring on resume. I think it is done this way so that in checkpoint type saves the resume of the original domain after checkpointing is cheap because nothing needs to be done.> And, about grant table, if we effectively suspend and resume PV driver, I > think we don''t need to migrate grant table entires. Is this right?Correct. Ian.