In testing save+restore on the gplpv drivers I''ve clocked up a pretty high domid - the domain I just created is id = 3015. I''ve noticed that my domains are taking longer and longer to create, eg there is a noticeable pause now between ''xm create'' and being able to connect via vnc, and I''m sure it was almost instant straight after Dom0 was last rebooted. To test if there was any cruft built up in my xenstore, I did a ''xenstore-ls | wc -l'', and got a segfault and a "kernel: invalid opcode: 0000 [2] SMP" message in my syslog. /var/lib/xenstored/tdb, which I assume is xenstore''s database, is 12730368 bytes in size. Is someone able to reproduce this problem? I''m using the xen-3.2-1 binaries from Debian Lenny. Thanks James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
James Harper writes ("[Xen-devel] Bug in xenstore once domid gets
big?"):> To test if there was any cruft built up in my xenstore, I did a
> ''xenstore-ls | wc -l'', and got a segfault and a
"kernel: invalid opcode:
> 0000 [2] SMP" message in my syslog.
Try
xenstore-ls -f -s
-s makes it use a socket rather than Xen IDC mechanisms to talk to
xenstore.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
> James Harper writes ("[Xen-devel] Bug in xenstore once domid getsbig?"):> > To test if there was any cruft built up in my xenstore, I did a > > ''xenstore-ls | wc -l'', and got a segfault and a "kernel: invalidopcode:> > 0000 [2] SMP" message in my syslog. > > Try > xenstore-ls -f -s > > -s makes it use a socket rather than Xen IDC mechanisms to talk to > xenstore. >That works, so I can see that there are 77949 lines in the xenstore when the domid counter is up to 3050. In comparison, a dom0 where the domid counter is up to 868 only has 1688 lines. Based on that very limited measurement, the rate of growth does not appear to be linear... The performance is definitely getting worse and worse the higher domid gets. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Try > > xenstore-ls -f -s > > > > -s makes it use a socket rather than Xen IDC mechanisms to talk to > > xenstore. > > > > That works, so I can see that there are 77949 lines in the xenstorewhen> the domid counter is up to 3050.It did work, but then I started getting connection refused, and things like ''xm list'' stopped working. I restarted xend but now all I get is: Domain-0 0 1001 1 r----- 301764.9 Domain-Unnamed 1 511 1 -b---- 27390.5 Even though there are at least 8 domains running. Time for a reboot I think. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
James Harper writes ("RE: [Xen-devel] Bug in xenstore once domid gets
big?"):> That works, so I can see that there are 77949 lines in the xenstore when
> the domid counter is up to 3050. In comparison, a dom0 where the domid
> counter is up to 868 only has 1688 lines. Based on that very limited
> measurement, the rate of growth does not appear to be linear...
Obviously there is some kind of leak. 77949 lines of what ?
Perhaps you could cut-and-paste a fragment.
> The performance is definitely getting worse and worse the higher domid
> gets.
The performance will be very bad if xenstore is this big.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
> James Harper writes ("RE: [Xen-devel] Bug in xenstore once domid gets > big?"): > > That works, so I can see that there are 77949 lines in the xenstorewhen> > the domid counter is up to 3050. In comparison, a dom0 where thedomid> > counter is up to 868 only has 1688 lines. Based on that very limited > > measurement, the rate of growth does not appear to be linear... > > Obviously there is some kind of leak. 77949 lines of what ? > Perhaps you could cut-and-paste a fragment. >If I''d received this email about 10 minutes ago I could have :( Xenstored crashed and so I''m going to have to reboot as the entire xenstore database got wiped... any way to reconstruct that? Oh well... that''s what test machines are for. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
James Harper writes ("RE: [Xen-devel] Bug in xenstore once domid gets
big?"):> [Ian:]
> > Obviously there is some kind of leak. 77949 lines of what ?
> > Perhaps you could cut-and-paste a fragment.
>
> If I''d received this email about 10 minutes ago I could have :(
So I saw.
> Oh well... that''s what test machines are for.
Hopefully you can reproduce the problem simply by creating and
destroying half a dozen or so test domains ? The xenstore-ls output
after that ought to make the leak clear.
> Xenstored crashed and so I''m going to have to reboot as the entire
> xenstore database got wiped... any way to reconstruct that?
I don''t know what version of tools and hypervisor you''re using
but
there have been some problems with overlong messages causing
misbehaviour. If any of these remain in xen-unstable we should fix
them, so it would be good to know what you were using ...
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Pascal Bouchareine
2008-Aug-14 14:33 UTC
Re: [Xen-devel] Bug in xenstore once domid gets big?
On Mon, Jul 28, 2008 at 02:33:47PM +0100, Ian Jackson wrote:> > > Obviously there is some kind of leak. 77949 lines of what ? > > > Perhaps you could cut-and-paste a fragment.> Hopefully you can reproduce the problem simply by creating and > destroying half a dozen or so test domains ? The xenstore-ls output > after that ought to make the leak clear.I think we have a leak when a domU crashes (didn''t search where yet) or restarts. I''ve been adding a _removeVm() in XendDomainInfo._restart to remove some entries and minimize the problem but it still exists a bit in the case of a domain crashing (during boot, for example) and being restarted for ever. -- \o/ Pascal Bouchareine - Gandi g 0170393757 15, place de la Nation - 75011 Paris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > On Mon, Jul 28, 2008 at 02:33:47PM +0100, Ian Jackson wrote: > > > > Obviously there is some kind of leak. 77949 lines of what ? > > > > Perhaps you could cut-and-paste a fragment. > > > Hopefully you can reproduce the problem simply by creating and > > destroying half a dozen or so test domains ? The xenstore-ls output > > after that ought to make the leak clear. > > I think we have a leak when a domU crashes (didn''t search where yet) > or restarts.I certainly had a lot of crashes in my journey to a domid > 3000. Mostly though, they were save + restore cycles. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel