Is analysis_phase just for debug purpose, since ''runs'' is just zero when invoked? Where is the possible place to mark pfn_type to as some special type for hvm domain, like invalid pfn? if ( hvm ) pfn_type[batch] = n; else pfn_type[batch] = pfn_to_mfn(n); I didn''t find code to check mmio holes. Then how does restore side differentiate normal ram from those holes, for a hvm with>4G memory?Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 17/11/08 06:50, "Tian, Kevin" <kevin.tian@intel.com> wrote:> I didn''t find code to check mmio holes. Then how does restore > side differentiate normal ram from those holes, for a hvm with >> 4G memory?I don''t think the save side transmits entries for non-existent pages. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >Sent: Monday, November 17, 2008 3:45 PM > >On 17/11/08 06:50, "Tian, Kevin" <kevin.tian@intel.com> wrote: > >> I didn''t find code to check mmio holes. Then how does restore >> side differentiate normal ram from those holes, for a hvm with >>> 4G memory? > >I don''t think the save side transmits entries for non-existent pages. >Yes, that''s also my original assumption. But after reading code, I didn''t find lines to seperate those non-existence pages. I saw one of your change (18533) to skip page patches which contain no valid pages: + if ( hvm ) + { + /* Look for and skip completely empty batches. */ + for ( j = 0; j < batch; j++ ) + if ( (pfn_type[j] & XEN_DOMCTL_PFINFO_LTAB_MASK) !+ XEN_DOMCTL_PFINFO_XTAB ) + break; + if ( j == batch ) + { + munmap(region_base, batch*PAGE_SIZE); + continue; /* bail on this batch: no valid pages */ + } + } + else But I don''t know where XEN_DOMCTL_PFINFO_XTAB may be set for hvm domain. Could you help spot it to me? Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 17/11/08 07:49, "Tian, Kevin" <kevin.tian@intel.com> wrote:> But I don''t know where XEN_DOMCTL_PFINFO_XTAB may be > set for hvm domain. Could you help spot it to me?xc_map_foreign_batch() -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >Sent: Monday, November 17, 2008 4:40 PM >On 17/11/08 07:49, "Tian, Kevin" <kevin.tian@intel.com> wrote: > >> But I don''t know where XEN_DOMCTL_PFINFO_XTAB may be >> set for hvm domain. Could you help spot it to me? > >xc_map_foreign_batch() > > -- KeirGot it. Thank you. Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir,
We are trying to fix a HVM live migration bug, and found "Use main memory
for
video memory" in r18383 cause guest hang after restore.
Following changes make all valid pages migrated, including vlapic page
(0xFEE00), and share page(0xFFFFF), so an extra memory population for these 2
pages would override previous mapping then cause guest hang (if using vlapic
acceleration).
What do you think of possible fixing? Skipping these specific pages in
xc_domain_save except video memory, or change the HVM domain creation.
Thanks,
====================================================================diff -r
2397555ebcc2 -r dade7f0bdc8d tools/libxc/xc_domain_save.c
--- a/tools/libxc/xc_domain_save.c Wed Aug 27 13:31:01 2008 +0100
+++ b/tools/libxc/xc_domain_save.c Wed Aug 27 14:53:39 2008 +0100
@@ -1109,12 +1109,6 @@ int xc_domain_save(int xc_handle, int io
if ( !((test_bit(n, to_send) && !test_bit(n, to_skip))
||
(test_bit(n, to_send) && last_iter) ||
(test_bit(n, to_fix) && last_iter)) )
- continue;
-
- /* Skip PFNs that aren''t really there */
- if ( hvm && ((n >= 0xa0 && n < 0xc0) /*
VGA hole */
- || (n >= (HVM_BELOW_4G_MMIO_START >>
PAGE_SHIFT)
- && n < (1ULL<<32) >>
PAGE_SHIFT)) /* MMIO */ )
continue;
/*
On Mon, Nov 17, 2008 at 04:40:19PM +0800, Keir Fraser
wrote:>
>
>
> On 17/11/08 07:49, "Tian, Kevin" <kevin.tian@intel.com>
wrote:
>
> > But I don''t know where XEN_DOMCTL_PFINFO_XTAB may be
> > set for hvm domain. Could you help spot it to me?
>
> xc_map_foreign_batch()
>
> -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
--
best rgds,
edwin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 18/11/08 01:17, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:> We are trying to fix a HVM live migration bug, and found "Use main memory for > video memory" in r18383 cause guest hang after restore. > > Following changes make all valid pages migrated, including vlapic page > (0xFEE00), and share page(0xFFFFF), so an extra memory population for these 2 > pages would override previous mapping then cause guest hang (if using vlapic > acceleration). > > What do you think of possible fixing? Skipping these specific pages in > xc_domain_save except video memory, or change the HVM domain creation.Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) in mod_l1_entry() to: mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt); if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) ) .... This will prevent foreign mappings of ''mmio'' pages. Pretty sensible and will fix this problem. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> On 18/11/08 01:17, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: > > >> We are trying to fix a HVM live migration bug, and found "Use main memory for >> video memory" in r18383 cause guest hang after restore. >> >> Following changes make all valid pages migrated, including vlapic page >> (0xFEE00), and share page(0xFFFFF), so an extra memory population for these 2 >> pages would override previous mapping then cause guest hang (if using vlapic >> acceleration). >> >> What do you think of possible fixing? Skipping these specific pages in >> xc_domain_save except video memory, or change the HVM domain creation. >> > > Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) in mod_l1_entry() > to: > mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt); > if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) ) > .... > > This will prevent foreign mappings of ''mmio'' pages. Pretty sensible and will > fix this problem. >It''s great! Does p2m_is_ram return 1 for video memory? I''ll make a small patch and test it. Thanks,> -- Keir > > >-- best rgds, edwin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/11/08 08:22, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:>> Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) in mod_l1_entry() >> to: >> mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt); >> if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) ) >> .... >> >> This will prevent foreign mappings of ''mmio'' pages. Pretty sensible and will >> fix this problem. >> > > It''s great! > Does p2m_is_ram return 1 for video memory?Yes, since it is just ordinary guest memory. That was the point of Samuel''s patch. -- Keir> I''ll make a small patch and test it._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir, As my test, video mem and vlapic page are okay, but share page is still treated as ram. After going through the source code, I think this should be "no harm". Although xc_domain_restore populate a new share page first, PV driver in HVM guest will remap share page when resume, thus destroying previous mapping and releasing previous page. So we want to use this simple patch, or mark the share page "not ram"? Keir Fraser wrote:> On 18/11/08 08:22, "Zhai, Edwin" <edwin.zhai@intel.com> wrote: > > >> Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) inmod_l1_entry()> >> to: > >> mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt); > >> if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) ) > >> .... > >> > >> This will prevent foreign mappings of ''mmio'' pages. Pretty sensibleand will> >> fix this problem. > >> > > It''s great! > > Does p2m_is_ram return 1 for video memory? > > Yes, since it is just ordinary guest memory. That was the point ofSamuel''s> patch. > > -- Keir > > > I''ll make a small patch and test it. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >-- best rgds, edwin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/11/08 09:35, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:> Keir, > As my test, video mem and vlapic page are okay, but share page is > still treated as ram. > > After going through the source code, I think this should be "no harm". > Although xc_domain_restore populate a new share page first, PV driver > in HVM guest will remap share page when resume, thus destroying > previous mapping and releasing previous page. > > So we want to use this simple patch, or mark the share page "not ram"?It''s fine. The shared page mapping is not set up until after normal guest memory is restored. So it''s safe. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel