Christopher S. Aker
2005-Apr-05 05:14 UTC
[Xen-devel] [BUG] memory leak in xend, console output
2.05 stable tools My xend process is currently at 462M. To reproduce: start a domain, telnet/xencons into the console, make some output (find / is a good way) and watch xend''s memory rise. Can''t restart xend, since it refuses to restart if there are domUs running (check this post) http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00051.html Thanks, -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> My xend process is currently at 462M. To reproduce: start a domain, telnet/xencons > into the console, make some output (find / is a good way) and watch xend''s memory > rise.Ok, it''s not console output. xend''s memory flickers up/down, but ends on the same number. I''m trying "watch -n0 xm list" now. So far, it''s gone up 700k in the last minute. It appears as though every run through "xm list" list adds ~2k -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m trying "watch -n0 xm list" now. So far, it''s gone up 700k in the > last minute. It appears as though every run through "xm list" list adds > ~2kI can replicate this too, so it does need looking into. I can''t really see what could be wrong with the code, though so would appreciate if someone else could poke around in there. Can you post a link to the graph you mentioned in #xen? Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > I''m trying "watch -n0 xm list" now. So far, it''s gone up 700k in the > > last minute. It appears as though every run through "xm list" list adds > > ~2k > > I can replicate this too, so it does need looking into. I can''t really see > what could be wrong with the code, though so would appreciate if someone > else could poke around in there. > > Can you post a link to the graph you mentioned in #xen?http://www.theshore.net/~caker/xen/memory-week.png The flat spot towards the end is from when I stopped running the above command. Thanks for looking into this! -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker wrote:>http://www.theshore.net/~caker/xen/memory-week.png > >The flat spot towards the end is from when I stopped running the above command. >Thanks for looking into this! > >I found this article while Googling. http://evanjones.ca/python-memory.html It specifically calls out Twisted as being known to cause this sort of behavior in the Python VM. For what it''s worth, I''ve been able to reproduce this memory leakage also with Xend. A workaround for current systems could be a cron job that restarts Xend every few days. Regards, Anthony Liguori>-Chris > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > A workaround for current systems could be a cron job that restarts Xend > every few days. >See other posts... xend can''t be restarted for many users. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> A workaround for current systems could be a cron job that restarts Xend > every few days.Xend won''t restart in my environment. The work-around that''s been posted does allow xend to be restarted, but not quite. It loses the naming of domains (which, I suppose is to be expected) but other nasty things happen that make it not a reliable solution for me. -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 7 Apr 2005, Christopher S. Aker wrote:> > A workaround for current systems could be a cron job that restarts Xend > > every few days. > > Xend won''t restart in my environment. The work-around that''s been posted does allow > xend to be restarted, but not quite. It loses the naming of domains (which, I > suppose is to be expected) but other nasty things happen that make it not a reliable > solution for me.Note, that this problaby isn''t that bad of a problem in the real world. We have a xen box with 4g and 20 domains(including dom0). 64g ram for dom0(128m swap). It''s been up for 84 days, and python only has 15m in use for xend. My guess is that as long as you don''t do a lot with xm/xend, the memory usage won''t grow. ps: the above box is running 2 tinderboxes constantly in an instance, and running a busy mailserver, all over nfsroot. It''s been rock solid(except for nfs issues, which aren''t xen related). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Note, that this problaby isn''t that bad of a problem in the real world. We > have a xen box with 4g and 20 domains(including dom0). 64g ram for dom0(128m > swap). It''s been up for 84 days, and python only has 15m in use for xend. > > My guess is that as long as you don''t do a lot with xm/xend, the memory usage > won''t grow.This happens under real-world usage, as well. One of our management scripts polls "xm list" every minute to gather statistics and state of the domains. The box has been up for 24 hours, and already xend is using > 120M. I could see other people doing something similar very easily and I hope that this problem isn''t just dismissed as a corner case. I guess it''s time to look towards the alternative management tools. Any chance of making them work with xen-2.0? Thanks, -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel