With 3.1.2rc2: bash-3.00# xm destroy sxc15 bash-3.00# xm list Name ID Mem VCPUs State Time(s) Domain-0 0 6596 4 r----- 202.7 sxc13 1024 1 52.0 sxc15 6 1024 1 ------ 0.0 It''s the below changeset - no code to reset the domid exists any more. It seems suboptimal to me? Shouldn''t refreshShutdown (or somewhere, anyway) remove domid? I noticed this whilst having another, more serious problem: sometimes, domains are never released, with a page still referenced (presumably?):>>> xc.domain_getinfo(21, 1)[{''paused'': 0, ''cpu_time'': 4895498373L, ''ssidref'': 0, ''hvm'': 0, ''shutdown_reason'': 0, ''dying'': 1, ''mem_kb'': 4L, ''domid'': 22, ''max_vcpu_id'': 0, ''crashed'': 0, ''running'': 0, ''maxmem_kb'': 1048576L, ''shutdown'': 0, ''online_vcpus'': 1, ''handle'': [75, 158, 234, 68, 113, 241, 195, 238, 113, 125, 31, 198, 154, 74, 44, 188], ''blocked'': 1}] None of the user-space daemons seem to be the culprit. I''m still trying to find out why xen thinks a page is still held. Note that this domain is still using flip instead of copy - I know that''s not the default, could a bug there have crept in? cheers john changeset: 15142:78389dbb08bb user: Steven Hand <steven@xensource.com> date: Fri Sep 14 16:33:34 2007 +0100 files: tools/python/xen/xend/XendDomainInfo.py description: This patch fixes the issue with waiting for devices to disconnect during the end stage of migration in Xend. The problem was cause by a deletion of the VM domain ID which is used by testDevicecompleComplete ->deviceIDs->backendRoot() to get the virtual backend device path. The virtual backend device path is used to check if a device still exists in xenstore. Signed-off-by: Yung Giang <yung.giang@gmail.com> xen-unstable changeset: 15128:f6928d6369999cd063edd361d592579c2483196b xen-unstable date: Tue May 22 15:32:50 2007 +0100 diff -r ac4a34b42602 -r 78389dbb08bb tools/python/xen/xend/XendDomainInfo.py --- a/tools/python/xen/xend/XendDomainInfo.py Wed May 16 10:42:07 2007 -0600 +++ b/tools/python/xen/xend/XendDomainInfo.py Fri Sep 14 16:33:34 2007 +0100 @@ -1684,7 +1684,6 @@ class XendDomainInfo: try: if self.domid is not None: xc.domain_destroy(self.domid) - self.domid = None for state in DOM_STATES_OLD: self.info[state] = 0 self._stateSet(DOM_STATE_HALTED) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 9/11/07 04:49, "John Levon" <levon@movementarian.org> wrote:> It''s the below changeset - no code to reset the domid exists any more. > It seems suboptimal to me? Shouldn''t refreshShutdown (or somewhere, > anyway) remove domid?Does this only happen for your special screwed domains that don''t actually die? You can use the ''q'' debug key to Xen to get it to print out per-domain info, including a list of held pages (if there''s only one or two pages held) and their reference counts. -- Keir> I noticed this whilst having another, more serious problem: sometimes, > domains are never released, with a page still referenced (presumably?): > >>>> xc.domain_getinfo(21, 1) > [{''paused'': 0, ''cpu_time'': 4895498373L, ''ssidref'': 0, ''hvm'': 0, > ''shutdown_reason'': 0, ''dying'': 1, ''mem_kb'': 4L, ''domid'': 22, > ''max_vcpu_id'': 0, ''crashed'': 0, ''running'': 0, ''maxmem_kb'': 1048576L, > ''shutdown'': 0, ''online_vcpus'': 1, ''handle'': [75, 158, 234, 68, 113, 241, > 195, 238, 113, 125, 31, 198, 154, 74, 44, 188], ''blocked'': 1}] > > None of the user-space daemons seem to be the culprit. I''m still trying > to find out why xen thinks a page is still held. Note that this domain > is still using flip instead of copy - I know that''s not the default, > could a bug there have crept in?_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Nov 09, 2007 at 07:42:33AM +0000, Keir Fraser wrote:> > It''s the below changeset - no code to reset the domid exists any more. > > It seems suboptimal to me? Shouldn''t refreshShutdown (or somewhere, > > anyway) remove domid? > > Does this only happen for your special screwed domains that don''t actually > die?I remember it being both, but it might have been - unfortunately I can only create screwed domains at the moment (!).> You can use the ''q'' debug key to Xen to get it to print out per-domain info, > including a list of held pages (if there''s only one or two pages held) and > their reference counts.Thanks! (XEN) Memory pages belonging to domain 1: (XEN) DomPage 000000019ddbf000: mfn=000000000019ddbf, caf=00000001, taf=0000000080000001 (XEN) Memory pages belonging to domain 2: (XEN) DomPage 00000001f4dbc000: mfn=00000000001f4dbc, caf=00000001, taf=0000000080000001 #define PGT_l4_page_table (4UL<<29) /* using this page as an L4 page table? */ Is it possible we do something unusual, and there''s an accounting bug? It seems that vcpu_destroy_pagetables() should kill any active reference. If I boot into the kernel debugger (so no userspace) and destroy the domain, it still happens. Before I try and work up something to track references to the kernel''s CR3 dompage, any suggestions or ideas? thanks john PS what''s the difference between PGT_base/root_page_table? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Oh dear. Do you map top-level pagetables into each other (i.e., linear mappings of other page tables)? Then you could have a circular reference where one level-4 pagetable references the other, and vice versa. But we though of that and break loops in relinquish_memory() in arch/x86/domain.c. Perhaps that''s broken though -- it''s certainly been modified recently to support linear mappings at lower levels in the pagetable hierarchy. I would add some tracing to the ''q'' key to dump the contents of the remaining few pages, and see if they actually reference each other. Also some tracing around relinquish_memory() is probably not a bad idea. -- Keir On 9/11/07 16:06, "John Levon" <levon@movementarian.org> wrote:> Thanks! > > (XEN) Memory pages belonging to domain 1: > (XEN) DomPage 000000019ddbf000: mfn=000000000019ddbf, caf=00000001, > taf=0000000080000001 > (XEN) Memory pages belonging to domain 2: > (XEN) DomPage 00000001f4dbc000: mfn=00000000001f4dbc, caf=00000001, > taf=0000000080000001 > > #define PGT_l4_page_table (4UL<<29) /* using this page as an L4 page table? > */ > > Is it possible we do something unusual, and there''s an accounting bug? It > seems > that vcpu_destroy_pagetables() should kill any active reference. If I boot > into > the kernel debugger (so no userspace) and destroy the domain, it still > happens. > > Before I try and work up something to track references to the kernel''s > CR3 dompage, any suggestions or ideas?_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, Nov 10, 2007 at 09:46:46AM +0000, Keir Fraser wrote:> Oh dear. Do you map top-level pagetables into each other (i.e., linear > mappings of other page tables)? Then you could have a circular referenceWe don''t (I''m told).> I would add some tracing to the ''q'' key to dump the contents of the > remaining few pages, and see if they actually reference each other. Also > some tracing around relinquish_memory() is probably not a bad idea.Note that we have exactly one page left, not some. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon writes ("Re: [Xen-devel] 15142:78389dbb08bb and domain
state"):> Note that we have exactly one page left, not some.
This may seem like a silly suggestion, but perhaps you should get a
copy of the data in this page and eyeball it. If you (or we) had an
idea what it looked like someone might be able to guess where it came
from.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Fri, Nov 09, 2007 at 04:06:31PM +0000, John Levon wrote:> > > It''s the below changeset - no code to reset the domid exists any more. > > > It seems suboptimal to me? Shouldn''t refreshShutdown (or somewhere, > > > anyway) remove domid? > > > > Does this only happen for your special screwed domains that don''t actually > > die? > > I remember it being both, but it might have been - unfortunately I can > only create screwed domains at the moment (!).It''s both: bash-3.00# xm list Name ID Mem VCPUs State Time(s) Domain-0 0 6718 4 r----- 119.7 sxc13 1024 1 52.0 sxc15 1 900 1 ------ 37.8 The domain is not listed in the ''q'' output at this point. Is there another way to fix the bug that regressed this behaviour? regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13/11/07 19:37, "John Levon" <levon@movementarian.org> wrote:>> I remember it being both, but it might have been - unfortunately I can >> only create screwed domains at the moment (!). > > It''s both: > > bash-3.00# xm list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 6718 4 r----- 119.7 > sxc13 1024 1 52.0 > sxc15 1 900 1 ------ 37.8 > > The domain is not listed in the ''q'' output at this point. Is there another way > to fix the bug that regressed this behaviour?Is it explicable how this happens for you but not for me? I haven''t been seeing domains hang around in ''cm list'' after domain destruction at all. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Nov 13, 2007 at 07:45:38PM +0000, Keir Fraser wrote:> >> I remember it being both, but it might have been - unfortunately I can > >> only create screwed domains at the moment (!). > > > > It''s both: > > > > bash-3.00# xm list > > Name ID Mem VCPUs State > > Time(s) > > Domain-0 0 6718 4 r----- 119.7 > > sxc13 1024 1 52.0 > > sxc15 1 900 1 ------ 37.8 > > > > The domain is not listed in the ''q'' output at this point. Is there another way > > to fix the bug that regressed this behaviour? > > Is it explicable how this happens for you but not for me? I haven''t been > seeing domains hang around in ''cm list'' after domain destruction at all.I presume that there''s something slightly different due to being Solaris dom0/domU. I would presume it''s our bug except that we don''t have changes to this area of the code, and by inspection I can''t see why anything would be different. One notable point is that this domU is still not using hvcopy for its networking. I still haven''t figured out the exact path: destroyDomain() is called, but something goes wrong later on regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Nov 16, 2007 at 12:01:38AM +0000, Keir Fraser wrote:> Reference counting for l4 pagetable entries happens in > get_page_from_l4e(),The circular reference was a red herring - I was looking at Xen''s own mapping in its private PTEs (d''oh). However, I did notice that we leak a ref count per VCPU, which gave me a big clue. Changeset 13302:7c5eea5feebd from Jan added this code: +#ifdef __x86_64__ + if ( pfn == pagetable_get_pfn(v->arch.guest_table_user) ) + v->arch.guest_table_user = pagetable_null(); +#endif Presumably for the benefit of compat guests where this is always true. However, it can be true on native Solaris guests too - when we switch to the kernel L4 we also load it into _USER_BASEPTR. The patch below fixes it for me (though I''m not positive it''s correct). Assuming this or something like it is the right thing, that will still leave the ''xm list'' problem, which I see with both Solaris and Linux domU''s. Still looking at that. cheers, john # HG changeset patch # User john.levon@sun.com # Date 1195467196 28800 # Node ID b972585a05838dc3a622438299572423a7a8de8d # Parent 00c6696dcc8b17cd9ea2325f260899e489e262e0 Fix VCPU pagetable cleanup Solaris can have the same PT in guest_table and guest_table_user without it being a compat domain. Be more careful when cleaning up. Signed-off-by: John Levon <john.levon@sun.com> diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c --- a/xen/arch/x86/domain.c +++ b/xen/arch/x86/domain.c @@ -1625,7 +1625,8 @@ static void vcpu_destroy_pagetables(stru else put_page_and_type(mfn_to_page(pfn)); #ifdef __x86_64__ - if ( pfn == pagetable_get_pfn(v->arch.guest_table_user) ) + if ( is_pv_32on64_vcpu(v) && + pfn == pagetable_get_pfn(v->arch.guest_table_user) ) v->arch.guest_table_user = pagetable_null(); #endif v->arch.guest_table = pagetable_null(); _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yeah, that''s a nasty one. I committed an alternative fix, which I think is clearer, as c/s 16401. Please give it a spin. I''ll also queue it for 3.1.3. -- Keir On 19/11/07 10:19, "John Levon" <levon@movementarian.org> wrote:> On Fri, Nov 16, 2007 at 12:01:38AM +0000, Keir Fraser wrote: > >> Reference counting for l4 pagetable entries happens in >> get_page_from_l4e(), > > The circular reference was a red herring - I was looking at Xen''s own > mapping in its private PTEs (d''oh). > > However, I did notice that we leak a ref count per VCPU, which gave me a > big clue. Changeset 13302:7c5eea5feebd from Jan added this code: > > +#ifdef __x86_64__ > + if ( pfn == pagetable_get_pfn(v->arch.guest_table_user) ) > + v->arch.guest_table_user = pagetable_null(); > +#endif > > Presumably for the benefit of compat guests where this is always true. > > However, it can be true on native Solaris guests too - when we switch > to the kernel L4 we also load it into _USER_BASEPTR. > > The patch below fixes it for me (though I''m not positive it''s correct). > Assuming this or something like it is the right thing, that will still > leave the ''xm list'' problem, which I see with both Solaris and Linux > domU''s. Still looking at that. > > cheers, > john > > # HG changeset patch > # User john.levon@sun.com > # Date 1195467196 28800 > # Node ID b972585a05838dc3a622438299572423a7a8de8d > # Parent 00c6696dcc8b17cd9ea2325f260899e489e262e0 > Fix VCPU pagetable cleanup > > Solaris can have the same PT in guest_table and guest_table_user without > it being a compat domain. Be more careful when cleaning up. > > Signed-off-by: John Levon <john.levon@sun.com> > > diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c > --- a/xen/arch/x86/domain.c > +++ b/xen/arch/x86/domain.c > @@ -1625,7 +1625,8 @@ static void vcpu_destroy_pagetables(stru > else > put_page_and_type(mfn_to_page(pfn)); > #ifdef __x86_64__ > - if ( pfn == pagetable_get_pfn(v->arch.guest_table_user) ) > + if ( is_pv_32on64_vcpu(v) && > + pfn == pagetable_get_pfn(v->arch.guest_table_user) ) > v->arch.guest_table_user = pagetable_null(); > #endif > v->arch.guest_table = pagetable_null();_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Nov 20, 2007 at 03:35:40PM +0000, Keir Fraser wrote:> Yeah, that''s a nasty one. I committed an alternative fix, which I think is > clearer, as c/s 16401. Please give it a spin.Looks good to me.> I''ll also queue it for 3.1.3.Thanks regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Nov 13, 2007 at 07:45:38PM +0000, Keir Fraser wrote:> > bash-3.00# xm list > > Name ID Mem VCPUs State > > Time(s) > > Domain-0 0 6718 4 r----- 119.7 > > sxc13 1024 1 52.0 > > sxc15 1 900 1 ------ 37.8 > > > > The domain is not listed in the ''q'' output at this point. Is there another way > > to fix the bug that regressed this behaviour? > > Is it explicable how this happens for you but not for me? I haven''t been > seeing domains hang around in ''cm list'' after domain destruction at all.Just to double-check, I''m testing with a managed domain, and not using xen-api for xm - same for you? I''m surprised it works every time for you, although since it appears to be a race, it''s possible (I suspect it might be because my domain is still page-flipping). Here''s what I see: We call destroyDomain(), and do: xc.domain_destroy(self.domid) for state in DOM_STATES_OLD: self.info[state] = 0 Now, this has attempted to force the states to be zero, but then something comes along and does XenDomain:_refresh(): running = self._running_domains() for dom in running: domid = dom[''domid''] if domid in self.domains: self.domains[domid].update(dom, refresh_shutdown) This immediately gets an up-to-date hypervisor version of the state, over-riding the forced state setting above. Previously, we''d delete the domid, so we wouldn''t get this update happening, as can be seen. I haven''t quite worked out when domID gets cleaned out when this race doesn''t happen though - what code resets domid in your case? Any suggestions for a fix? regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 22/11/07 15:07, "John Levon" <levon@movementarian.org> wrote:>> Is it explicable how this happens for you but not for me? I haven''t been >> seeing domains hang around in ''cm list'' after domain destruction at all. > > Just to double-check, I''m testing with a managed domain, and not using > xen-api for xm - same for you?That could be the difference. I don''t think I''ve ever run a managed domain. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 22/11/07 15:13, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:>>> Is it explicable how this happens for you but not for me? I haven''t been >>> seeing domains hang around in ''cm list'' after domain destruction at all. >> >> Just to double-check, I''m testing with a managed domain, and not using >> xen-api for xm - same for you? > > That could be the difference. I don''t think I''ve ever run a managed domain.I have now, and it works for me. My domid gets zapped in cleanupDomain(). It gets called twice from destroyDomain(), once via XendDomain.instance.remove_domain() and the other as a direct invocation. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Nov 22, 2007 at 03:52:41PM +0000, Keir Fraser wrote:> >> Just to double-check, I''m testing with a managed domain, and not using > >> xen-api for xm - same for you? > > > > That could be the difference. I don''t think I''ve ever run a managed domain. > > I have now, and it works for me. My domid gets zapped in cleanupDomain(). It > gets called twice from destroyDomain(), once via > XendDomain.instance.remove_domain() and the other as a direct invocation.I don''t see any code to zap domid there. BTW the call from remove_domain() shouldn''t happen as the _stateGet will be DOM_STATE_HALTED at that point. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> I have now, and it works for me. My domid gets zapped in cleanupDomain(). It >> gets called twice from destroyDomain(), once via >> XendDomain.instance.remove_domain() and the other as a direct invocation. > > I don''t see any code to zap domid there. BTW the call from > remove_domain() shouldn''t happen as the _stateGet will be > DOM_STATE_HALTED at that point.Looks like xen-unstable c/s 15587 needs backporting to 3.1 branch. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Nov 22, 2007 at 04:07:55PM +0000, Keir Fraser wrote:> >> I have now, and it works for me. My domid gets zapped in cleanupDomain(). It > >> gets called twice from destroyDomain(), once via > >> XendDomain.instance.remove_domain() and the other as a direct invocation. > > > > I don''t see any code to zap domid there. BTW the call from > > remove_domain() shouldn''t happen as the _stateGet will be > > DOM_STATE_HALTED at that point. > > Looks like xen-unstable c/s 15587 needs backporting to 3.1 branch.Thanks! It does indeed work. However, isn''t it effectively regressing the bug fixed by 15142:78389dbb08bb, whatever that is? It just moves the clearing of domid a couple of lines down. regards, john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 22/11/07 16:44, "John Levon" <levon@movementarian.org> wrote:>> Looks like xen-unstable c/s 15587 needs backporting to 3.1 branch. > > Thanks! It does indeed work. > > However, isn''t it effectively regressing the bug fixed by > 15142:78389dbb08bb, whatever that is? It just moves the clearing of > domid a couple of lines down.Good question. I think that would need further investigation. Perhaps cleanupDomain should only zap the domid in certain situations? I guess we''ll revisit if we ever see the bug claimed in 15142. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Nov 22, 2007 at 05:32:35PM +0000, Keir Fraser wrote:> >> Looks like xen-unstable c/s 15587 needs backporting to 3.1 branch. > > > > Thanks! It does indeed work. > > > > However, isn''t it effectively regressing the bug fixed by > > 15142:78389dbb08bb, whatever that is? It just moves the clearing of > > domid a couple of lines down. > > Good question. I think that would need further investigation. Perhaps > cleanupDomain should only zap the domid in certain situations? I guess we''ll > revisit if we ever see the bug claimed in 15142.Cc:ing authors of 15142 to see if they can elucidate at all. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel