I inadvertently found a way to crash/hang xend and almost completely lock oneself out of a domain. I was testing a new domain, and without thinking I did ''cat /dev/urandom'' (rather than head or tail, oopsie) to make sure it was functioning. The resulting flood of output pegged xend at 99% cpu, where it stayed for 10 minutes before I killed it off. The real issue came after a restart, when I attempted to console back into the running domain. Either I had not managed to ctrl+c the `cat` prior to hanging xend or the output buffer was so full that it continued to spew garbage at me when I would restart xend and console in (I''m guessing). I got back on the the console a minute or so later by either managing to ctrl+c at the right time, or the output buffer finally emptied. This specific situation is not so scary since only people who didn''t sleep enough *cough* would be cat-ing /dev/urandom, but I found that the event is not limited to something that silly. It seems that any large output of binary to the console will hang xend. I can reproduce the exact situation with "cat /usr/bin/vim" or any other decent-sized binary. Xend managed to recover from cat-ing binaries up to about 1.5mb, but anything larger than 2mb had xend using 99% cpu anywhere from 5 minutes to indefinitely (longer than I could wait anyway). Memory usage during the hangs does not go up appreciably. Strategic ctrl+c gets me back to console with relative ease, but it still requires a xend restart (sometimes 2 or 3) before I can get back in. In contrast to the binary output, even extremely large text files (500mb+) appear to have no ill effect, even when cat''d in a while loop. I have not tried outside of my setup so I don''t know if it''s reproducable, but it is very concerning to me that any user in any domain could, with a typo, crush the CPU and take out all management capability as well. I''ll provide any additional info if it will help. Running Xen 2.0.3 Hardware: Dual Athlon XP 1900+ (1.6g) 512mb RAM Domain0: Gentoo 2004.3 running 2.6.10-xen0 128mb RAM allocated CPU0 DomainU: Gentoo 2004.3 2.6.10-xenU / default config / 128mb RAM allocated CPU1 -- +--------------------------------+ Mike Culbertson Senior Systems Administrator DiD : 646-230-8752/(M)646-382-4779 Mail: mike@infoleak.com +--------------------------------+ ------------------------------------------------------- This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting Tool for open source databases. Create drag-&-drop reports. Save time by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. Download a FREE copy at http://www.intelliview.com/go/osdn_nl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> It seems that any large > output of > binary to the console will hang xend. I can reproduce the > exact situation > with "cat /usr/bin/vim" or any other decent-sized binary. > Xend managed to > recover from cat-ing binaries up to about 1.5mb, but anything > larger than 2mb > had xend using 99% cpu anywhere from 5 minutes to > indefinitely (longer than I > could wait anyway).Yep, fixing this has been on the todo list for a while. I''m surprised its taken someone so long to spot it :-) We need to limit the rate at which xend is prepared to take message off a given control ring. There''ll then be back pressure to the domain, which can either choose to block or throw away messages.> In contrast to the binary output, even extremely large text > files (500mb+) > appear to have no ill effect, even when cat''d in a while loop.I think you''re just getting lucky. At least I can''t think why there''d be a difference between binary and text, unless xend does some scanning of the console messages I''m not aware of.> I have not tried outside of my setup so I don''t know if it''s > reproducable, but > it is very concerning to me that any user in any domain > could, with a typo, > crush the CPU and take out all management capability as well. > I''ll provide > any additional info if it will help.Yep, this is serious and needs fixing. Volunteers? Thanks, Ian ------------------------------------------------------- This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting Tool for open source databases. Create drag-&-drop reports. Save time by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. Download a FREE copy at http://www.intelliview.com/go/osdn_nl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Hi, On Thu, Jan 27, 2005 at 02:19:35AM -0800, Mike Culbertson wrote:> with "cat /usr/bin/vim" or any other decent-sized binary. Xend managed to > recover from cat-ing binaries up to about 1.5mb, but anything larger than 2mb > had xend using 99% cpu anywhere from 5 minutes to indefinitely (longer than I > could wait anyway). Memory usage during the hangs does not go up appreciably. > Strategic ctrl+c gets me back to console with relative ease, but it still > requires a xend restart (sometimes 2 or 3) before I can get back in.For the record, I have seen the same thing too. Running a simple "yes" command in a domain''s console has the same effect. Gabor ------------------------------------------------------- This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting Tool for open source databases. Create drag-&-drop reports. Save time by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. Download a FREE copy at http://www.intelliview.com/go/osdn_nl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel