Xen devel - May 2007 - Xrenstore and many save/restore cycles...

Let me first say that I have very little understanding of xenstore, as
it''s not an area of Xen that I normally work in. If I''ve
completely
missed something, please let me know.

I''ve been running a script over the (extended) weekend which saves and
restores HVM domains. In this case, two different domains. 

At this moment in time (around 6000 save/restore operations total),
xenstored is using about 900MB of memory (which is a bit excessive on my
256MB Dom0). Thrashing of the swap is consuming enough time that my
machine is unable to cope with running the domains satisfactorily at
this time. 

Using xenstored-list, I see hundreds (so far) of instance information of
"old" domains that have been killed when being saved, and created with
a
new ID when restored. So I''ve got (I''m guessing) some 6000 or
so domains
stored in xenstore. 

Is this something intended, and if so, would [I''m asking because I
haven''t tried it] the same effect be expected if 6000 domains where
created and destroyed? If so, what is the purpose of keeping information
about a domain that is no longer there?

I can certainly see problems with servers that for one reason or another
save/restore several domains daily - after a few months, there will be
LOTS of data in the xenstore and potential swap-thrashin causing
problems, or what???

--
Mats



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 9/5/07 17:09, "Petersson, Mats" <Mats.Petersson@amd.com>
wrote:
> Is this something intended, and if so, would [I''m asking because I
> haven''t tried it] the same effect be expected if 6000 domains
where
> created and destroyed? If so, what is the purpose of keeping information
> about a domain that is no longer there?
> 
> I can certainly see problems with servers that for one reason or another
> save/restore several domains daily - after a few months, there will be
> LOTS of data in the xenstore and potential swap-thrashin causing
> problems, or what???
These days xenstore only needs to hold information about live domains. The
/vm/<uuid> directory should be removed by xend once a domain is completely
dead. It''d be great to get a patch to fix this!

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> On 9/5/07 17:09, "Petersson, Mats" <Mats.Petersson@amd.com>
wrote:
> 
> > Is this something intended, and if so, would [I''m asking
because I
> > haven''t tried it] the same effect be expected if 6000 domains
where
> > created and destroyed? If so, what is the purpose of 
> keeping information
> > about a domain that is no longer there?
> > 
> > I can certainly see problems with servers that for one 
> reason or another
> > save/restore several domains daily - after a few months, 
> there will be
> > LOTS of data in the xenstore and potential swap-thrashin causing
> > problems, or what???
> 
> These days xenstore only needs to hold information about live 
> domains. The
> /vm/<uuid> directory should be removed by xend once a domain 
> is completely
> dead. It''d be great to get a patch to fix this!
Ok, I''m happy to make an attempt to fix this, but I haven''t
really got a
clue even where to start looking to find the right place to remove it...
Any hint?

--
Mats
> 
>  -- Keir
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 9/5/07 17:18, "Petersson, Mats" <Mats.Petersson@amd.com>
wrote:
>> These days xenstore only needs to hold information about live
>> domains. The
>> /vm/<uuid> directory should be removed by xend once a domain
>> is completely
>> dead. It''d be great to get a patch to fix this!
> 
> Ok, I''m happy to make an attempt to fix this, but I
haven''t really got a
> clue even where to start looking to find the right place to remove it...
> Any hint?
Hmmm... Somewhere in tools/python/xen/xend is the best I can do!

The /local/domain/foo directory seems to disappear nicely. Perhaps the
/vm/foo directory could go at the same time without things breaking?

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: Keir Fraser [mailto:keir@xensource.com] 
> Sent: 09 May 2007 17:46
> To: Petersson, Mats; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Xrenstore and many save/restore cycles...
> 
> On 9/5/07 17:18, "Petersson, Mats" <Mats.Petersson@amd.com>
wrote:
> 
> >> These days xenstore only needs to hold information about live
> >> domains. The
> >> /vm/<uuid> directory should be removed by xend once a domain
> >> is completely
> >> dead. It''d be great to get a patch to fix this!
> > 
> > Ok, I''m happy to make an attempt to fix this, but I
haven''t
> really got a
> > clue even where to start looking to find the right place to 
> remove it...
> > Any hint?
> 
> Hmmm... Somewhere in tools/python/xen/xend is the best I can do!
I started looking there while waiting for a reply - didn''t find
anything
real useful, but I will look again tomorrow unless someone comes up with
a solution before then. 
> 
> The /local/domain/foo directory seems to disappear nicely. Perhaps the
> /vm/foo directory could go at the same time without things breaking?
Will have a look at the /local/... Path and see if I can find where
that''s removed. So far I didn''t find any (particularly useful)
"xenstore
remove" operations.... 

--
Mats
> 
>  -- Keir
> 
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Petersson, Mats
> Sent: 09 May 2007 17:50
> To: Keir Fraser; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Xrenstore and many save/restore cycles...
> 
>  
> 
> > -----Original Message-----
> > From: Keir Fraser [mailto:keir@xensource.com] 
> > Sent: 09 May 2007 17:46
> > To: Petersson, Mats; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Xrenstore and many save/restore cycles...
> > 
> > On 9/5/07 17:18, "Petersson, Mats"
<Mats.Petersson@amd.com> wrote:
> > 
> > >> These days xenstore only needs to hold information about live
> > >> domains. The
> > >> /vm/<uuid> directory should be removed by xend once a
domain
> > >> is completely
> > >> dead. It''d be great to get a patch to fix this!
> > > 
> > > Ok, I''m happy to make an attempt to fix this, but I
haven''t
> > really got a
> > > clue even where to start looking to find the right place to 
> > remove it...
> > > Any hint?
> > 
> > Hmmm... Somewhere in tools/python/xen/xend is the best I can do!
> 
> I started looking there while waiting for a reply - didn''t 
> find anything
> real useful, but I will look again tomorrow unless someone 
> comes up with
> a solution before then. 
Well, I''ve looked at it further, and it''s not the
/local/domain/foo that
is left behind (or at least, that''s not the first bit that comes up).
> 
> > 
> > The /local/domain/foo directory seems to disappear nicely. 
> Perhaps the
> > /vm/foo directory could go at the same time without things breaking?
> 
> Will have a look at the /local/... Path and see if I can find where
> that''s removed. So far I didn''t find any (particularly 
> useful) "xenstore
> remove" operations.... 
> 
> --
> Mats
> > 
> >  -- Keir
> > 
> > 
> > 
> > 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com 
> > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> > Petersson, Mats
> > Sent: 09 May 2007 17:50
> > To: Keir Fraser; xen-devel@lists.xensource.com
> > Subject: RE: [Xen-devel] Xrenstore and many save/restore cycles...
> > 
> >  
> > 
> > > -----Original Message-----
> > > From: Keir Fraser [mailto:keir@xensource.com] 
> > > Sent: 09 May 2007 17:46
> > > To: Petersson, Mats; xen-devel@lists.xensource.com
> > > Subject: Re: [Xen-devel] Xrenstore and many save/restore
cycles...
> > > 
> > > On 9/5/07 17:18, "Petersson, Mats"
<Mats.Petersson@amd.com> wrote:
> > > 
> > > >> These days xenstore only needs to hold information about
live
> > > >> domains. The
> > > >> /vm/<uuid> directory should be removed by xend
once a domain
> > > >> is completely
> > > >> dead. It''d be great to get a patch to fix this!
> > > > 
> > > > Ok, I''m happy to make an attempt to fix this, but I
haven''t
> > > really got a
> > > > clue even where to start looking to find the right place to 
> > > remove it...
> > > > Any hint?
> > > 
> > > Hmmm... Somewhere in tools/python/xen/xend is the best I can do!
> > 
> > I started looking there while waiting for a reply - didn''t 
> > find anything
> > real useful, but I will look again tomorrow unless someone 
> > comes up with
> > a solution before then. 
> 
> Well, I''ve looked at it further, and it''s not the 
> /local/domain/foo that is left behind (or at least, that''s 
> not the first bit that comes up). 
Sorry about that, hit Send when I really didn''t want to do that... 

I''ve looked for /vm/foo, which is very short, so that''s not
what is
filling up the xenstore. 

It seems like I''ve got a lot of /local/domain/0/backend/console/* stuff
tho. Not sure if I have any other .../0/backend/* stuff as well, but
certainly there are literally thousands of consoles there.. 

Trying to locate where these are created and restored, but no success
yet. 

--
Mats
> > 
> > > 
> > > The /local/domain/foo directory seems to disappear nicely. 
> > Perhaps the
> > > /vm/foo directory could go at the same time without 
> things breaking?
> > 
> > Will have a look at the /local/... Path and see if I can find where
> > that''s removed. So far I didn''t find any
(particularly
> > useful) "xenstore
> > remove" operations.... 
> > 
> > --
> > Mats
> > > 
> > >  -- Keir
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> > 
> > 
> > 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10/5/07 14:47, "Petersson, Mats" <Mats.Petersson@amd.com>
wrote:
> It seems like I''ve got a lot of /local/domain/0/backend/console/*
stuff
> tho. Not sure if I have any other .../0/backend/* stuff as well, but
> certainly there are literally thousands of consoles there..
> 
> Trying to locate where these are created and restored, but no success
> yet. 
It may be that xend is deleting individual subdirectories of
/local/domain/foo rather than simply the whole thing. Even if we continue to
delete individual subdirectories, we should delete the directory itself at
some point during domain destruction in xend.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, May 10, 2007 at 03:47:33PM +0200, Petersson, Mats
wrote:
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xensource.com 
> > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> > > Petersson, Mats
> > > Sent: 09 May 2007 17:50
> > > To: Keir Fraser; xen-devel@lists.xensource.com
> > > Subject: RE: [Xen-devel] Xrenstore and many save/restore
cycles...
> > > 
> > >  
> > > 
> > > > -----Original Message-----
> > > > From: Keir Fraser [mailto:keir@xensource.com] 
> > > > Sent: 09 May 2007 17:46
> > > > To: Petersson, Mats; xen-devel@lists.xensource.com
> > > > Subject: Re: [Xen-devel] Xrenstore and many save/restore
cycles...
> > > > 
> > > > On 9/5/07 17:18, "Petersson, Mats"
<Mats.Petersson@amd.com> wrote:
> > > > 
> > > > >> These days xenstore only needs to hold information
about live
> > > > >> domains. The
> > > > >> /vm/<uuid> directory should be removed by
xend once a domain
> > > > >> is completely
> > > > >> dead. It''d be great to get a patch to fix
this!
> > > > > 
> > > > > Ok, I''m happy to make an attempt to fix this,
but I haven''t
> > > > really got a
> > > > > clue even where to start looking to find the right
place to
> > > > remove it...
> > > > > Any hint?
> > > > 
> > > > Hmmm... Somewhere in tools/python/xen/xend is the best I can
do!
> > > 
> > > I started looking there while waiting for a reply -
didn''t
> > > find anything
> > > real useful, but I will look again tomorrow unless someone 
> > > comes up with
> > > a solution before then. 
> > 
> > Well, I''ve looked at it further, and it''s not the 
> > /local/domain/foo that is left behind (or at least, that''s 
> > not the first bit that comes up). 
> 
> Sorry about that, hit Send when I really didn''t want to do that...
> 
> I''ve looked for /vm/foo, which is very short, so that''s
not what is
> filling up the xenstore. 
> 
> It seems like I''ve got a lot of /local/domain/0/backend/console/*
stuff
> tho. Not sure if I have any other .../0/backend/* stuff as well, but
> certainly there are literally thousands of consoles there.. 
> 
> Trying to locate where these are created and restored, but no success
> yet. 
If its the virtual framebuffer bits, then they''re populated in a couple
of places

tools/python/xen/xend/server/vfbif.py

sets up the basic node & permissions. While in PV case, the daemon like
/usr/lib/xen/bin/xen-vncfb (or xen-sdlfb) populates the rest. Can''t say
offhand where the info would be populated in the HVM case since this code
changed fairly recently & i''ve not checked up on its new workings.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

If you look in python/xen/xend/server at the various device  
controllers, like blkif.py etc they should do the tidying up.

Cheers

Tom

On 10 May 2007, at 14:47, Petersson, Mats wrote:
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com
>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of
>>> Petersson, Mats
>>> Sent: 09 May 2007 17:50
>>> To: Keir Fraser; xen-devel@lists.xensource.com
>>> Subject: RE: [Xen-devel] Xrenstore and many save/restore cycles...
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Keir Fraser [mailto:keir@xensource.com]
>>>> Sent: 09 May 2007 17:46
>>>> To: Petersson, Mats; xen-devel@lists.xensource.com
>>>> Subject: Re: [Xen-devel] Xrenstore and many save/restore
cycles...
>>>>
>>>> On 9/5/07 17:18, "Petersson, Mats"
<Mats.Petersson@amd.com> wrote:
>>>>
>>>>>> These days xenstore only needs to hold information
about live
>>>>>> domains. The
>>>>>> /vm/<uuid> directory should be removed by xend
once a domain
>>>>>> is completely
>>>>>> dead. It''d be great to get a patch to fix
this!
>>>>>
>>>>> Ok, I''m happy to make an attempt to fix this, but
I haven''t
>>>> really got a
>>>>> clue even where to start looking to find the right place to
>>>> remove it...
>>>>> Any hint?
>>>>
>>>> Hmmm... Somewhere in tools/python/xen/xend is the best I can
do!
>>>
>>> I started looking there while waiting for a reply - didn''t
>>> find anything
>>> real useful, but I will look again tomorrow unless someone
>>> comes up with
>>> a solution before then.
>>
>> Well, I''ve looked at it further, and it''s not the
>> /local/domain/foo that is left behind (or at least, that''s
>> not the first bit that comes up).
>
> Sorry about that, hit Send when I really didn''t want to do that...
>
> I''ve looked for /vm/foo, which is very short, so that''s
not what is
> filling up the xenstore.
>
> It seems like I''ve got a lot of /local/domain/0/backend/console/*
> stuff
> tho. Not sure if I have any other .../0/backend/* stuff as well, but
> certainly there are literally thousands of consoles there..
>
> Trying to locate where these are created and restored, but no success
> yet.
>
> --
> Mats
>>>
>>>>
>>>> The /local/domain/foo directory seems to disappear nicely.
>>> Perhaps the
>>>> /vm/foo directory could go at the same time without
>> things breaking?
>>>
>>> Will have a look at the /local/... Path and see if I can find where
>>> that''s removed. So far I didn''t find any
(particularly
>>> useful) "xenstore
>>> remove" operations....
>>>
>>> --
>>> Mats
>>>>
>>>>  -- Keir
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: Keir Fraser [mailto:keir@xensource.com] 
> Sent: 10 May 2007 14:56
> To: Petersson, Mats; Keir Fraser; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Xrenstore and many save/restore cycles...
> 
> 
> 
> 
> On 10/5/07 14:47, "Petersson, Mats"
<Mats.Petersson@amd.com> wrote:
> 
> > It seems like I''ve got a lot of 
> /local/domain/0/backend/console/* stuff
> > tho. Not sure if I have any other .../0/backend/* stuff as well, but
> > certainly there are literally thousands of consoles there..
> > 
> > Trying to locate where these are created and restored, but 
> no success
> > yet. 
> 
> It may be that xend is deleting individual subdirectories of
> /local/domain/foo rather than simply the whole thing. Even if 
> we continue to
> delete individual subdirectories, we should delete the 
> directory itself at
> some point during domain destruction in xend.
No, it''s not /local/domain/<n> that is the problem. 

It''s specifically the /local/domain/0/backend/console that is a
problem.
Trying to track down where it is at the moment. Will follow Tom Wilkies
hints to see where that leads. 

Tom, if you have any further comments, please let me know. 

--
Mats
> 
>  -- Keir
> 
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > It seems like I''ve got a lot of 
> /local/domain/0/backend/console/* stuff
> > tho. Not sure if I have any other .../0/backend/* stuff as well, but
> > certainly there are literally thousands of consoles there.. 
> > 
> > Trying to locate where these are created and restored, but 
> no success
> > yet. 
/s/restored/destroyed/ above.
> 
> If its the virtual framebuffer bits, then they''re populated 
> in a couple
> of places
> 
> tools/python/xen/xend/server/vfbif.py

Thanks Daniel, 

I don''t think it''s the vfb, as it''s called
"console" in the xenstore.

Attached is the diff (xs.diff) between an "early days" and
"current"
xenstore-ls output whilst running a save/restore script and the "early
days" output (xs.saved). [This is after a restart of the machine with
the latest xen-unstable (c/s 15017). I also cleaned out the original
database for xenstore, to be sure that all is "neat"]. 

--
Mats




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Petersson, Mats
> Sent: 10 May 2007 16:51
> To: Daniel P. Berrange
> Cc: xen-devel@lists.xensource.com; Keir Fraser
> Subject: RE: [Xen-devel] Xrenstore and many save/restore cycles...
> 
> > > It seems like I''ve got a lot of 
> > /local/domain/0/backend/console/* stuff
> > > tho. Not sure if I have any other .../0/backend/* stuff 
> as well, but
> > > certainly there are literally thousands of consoles there.. 
> > > 
> > > Trying to locate where these are created and restored, but 
> > no success
> > > yet. 
> 
> /s/restored/destroyed/ above.
> > 
> > If its the virtual framebuffer bits, then they''re populated 
> > in a couple
> > of places
> > 
> > tools/python/xen/xend/server/vfbif.py
> 
> 
> Thanks Daniel, 
> 
> I don''t think it''s the vfb, as it''s called
"console" in the xenstore.
> 
> Attached is the diff (xs.diff) between an "early days" and
"current"
> xenstore-ls output whilst running a save/restore script and the "early
> days" output (xs.saved). [This is after a restart of the machine with
> the latest xen-unstable (c/s 15017). I also cleaned out the original
> database for xenstore, to be sure that all is "neat"]. 
Also just noticed that this "growth" of the Xenstore happens on normal
"create/destroy" too.

Still trying to understand where these are supposed to be removed. 

--
Mats
> 
> --
> Mats
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: Petersson, Mats 
> Sent: 10 May 2007 17:30
> To: Petersson, Mats; Daniel P. Berrange
> Cc: xen-devel@lists.xensource.com; Keir Fraser
> Subject: RE: [Xen-devel] Xrenstore and many save/restore cycles...
> 
>  
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com 
> > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> > Petersson, Mats
> > Sent: 10 May 2007 16:51
> > To: Daniel P. Berrange
> > Cc: xen-devel@lists.xensource.com; Keir Fraser
> > Subject: RE: [Xen-devel] Xrenstore and many save/restore cycles...
> > 
> > > > It seems like I''ve got a lot of 
> > > /local/domain/0/backend/console/* stuff
> > > > tho. Not sure if I have any other .../0/backend/* stuff 
> > as well, but
> > > > certainly there are literally thousands of consoles there.. 
> > > > 
> > > > Trying to locate where these are created and restored, but 
> > > no success
> > > > yet. 
> > 
> > /s/restored/destroyed/ above.
> > > 
> > > If its the virtual framebuffer bits, then they''re
populated
> > > in a couple
> > > of places
> > > 
> > > tools/python/xen/xend/server/vfbif.py
> > 
> > 
> > Thanks Daniel, 
> > 
> > I don''t think it''s the vfb, as it''s called
"console" in the
> xenstore. 
> > 
> > Attached is the diff (xs.diff) between an "early days" and
"current"
> > xenstore-ls output whilst running a save/restore script and 
> the "early
> > days" output (xs.saved). [This is after a restart of the 
> machine with
> > the latest xen-unstable (c/s 15017). I also cleaned out the original
> > database for xenstore, to be sure that all is "neat"]. 
> 
> Also just noticed that this "growth" of the Xenstore happens 
> on normal "create/destroy" too.
> 
> Still trying to understand where these are supposed to be removed. 
As far as I can tell, the device should be removed around the place
where the domains device is removed - around line 1340 in
XendDomainInfo. But I''m still unsure about how this actually works. 

--
Mats
> 
> --
> Mats
> > 
> > --
> > Mats
> > 
> > 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel