-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello. I''m a new Xen user and I''m quite happy what I''ve seen so far. I''m migrating couple our servers to xen and I encountered this "funny" situation where xend crashes(?) but some of the domU''s are still up and running. It seems that since my dom0 doesn''t have really much memory xend gets killed by dom0 kernel due to filled memory. I''m not (yet) sure if this is the case, but it seems that way. I changed dom0-min-memory setting from 192 to 256, but since there''s a lots of data traffic currently from development services going to domU''s I didn''t reboot the machine yet. And secondly, after I restart xend by hand via init.d I don''t get proper output from xm list (I stripped output a bit): Name ID Mem Domain-0 0 491 Domain-Unnamed 1 128 There was 3 domU''s running when the xend died, two of them seems to be dead at the moment and third is running, which is shown on xm list, even tho ''Domain-Unnamed'' isn''t corrent name for the machine. Basically the question I''m after is that is it even possible (or perhaps even normal) that xend is killed by the kernel if memory runs out? And is it possible to recover domU''s and the rest of the system properly when xend dies on the fly? - -- Tapio Salonsaari My public GPG key is available at http://take.nerd.fi -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkove9EACgkQh/RMqfItS5WRUQCfWZDfpqYe19JmxPHg0kSJVp1j kKgAn2+fHCzDEDmCzRYTctSlay4R1DGF =+oEv -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 10/06/09 10:24, Tapio Salonsaari wrote:> Basically the question I''m after is that is it even possible (or perhaps > even normal) that xend is killed by the kernel if memory runs out? And > is it possible to recover domU''s and the rest of the system properly > when xend dies on the fly?Its certainly possible - in extreme low memory situations the Linux OOM killer will just look for a process consuming the most memory and destroy it to free up memory. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 10/06/09 11:39, Tapio Salonsaari wrote:> Ok, so it seems my assumptions was correct. I knew that it''s possible > that kernel will kill processes if there''s no resources to keep them > running but I just wondered if xend is somehow protected from it. I > don''t know if this is even possible but if it is IMO xend should > automatically have higher priority so that it''s not the first on line of > fire.Does your system have plenty of swap? I''ve been running a beefy dom0 with at least 20 odd domUs running and dom0 has only had a fixed 512MB allocated to it for the majority of its life.> Stability is however still an issue and I''m not yet sure if growing > dom0-min-memory -limit solved the problem. And I''m still interested why > some of domU''s survived xend crash and is it even possible to recover > from such situation gracefully?I''m not sure I can answer this bit for you as I''ve never really been in the situation.> - -- > Tapio Salonsaari > > My public GPG key is available > at http://take.nerd.fi > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAkovjU8ACgkQh/RMqfItS5VM4ACfXA/phQpL/qqZtN0McCmkUsKZ > wWwAnAk7Tc+0dwfyzo+2chOrirYByKcE > =2pQa > -----END PGP SIGNATURE------- Adam Wilbraham - Systems Administrator TechnoPhobia Limited The Workstation 15 Paternoster Row SHEFFIELD England S1 2BX t: +44 (0)114 2212123 f: +44 (0)114 2212124 e: adam.wilbraham@technophobia.com w: http://www.technophobia.com/ Registered in England and Wales Company No. 3063669 VAT registration No. 598 7858 42 ISO 9001:2000 Accredited Company No. 21227 ISO 14001:2004 Accredited Company No. E997 ISO 27001:2005 (BS7799) Accredited Company No. IS 508906 Investor in People Certified No. 101507 The contents of this email are confidential to the addressee and are intended solely for the recipients use. If you are not the addressee, you have received this email in error. Any disclosure, copying, distribution or action taken in reliance on it is prohibited and may be unlawful. Any opinions expressed in this email are those of the author personally and not TechnoPhobia Limited who do not accept responsibility for the contents of the message. All email communications, in and out of TechnoPhobia, are recorded for monitoring purposes. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Adam Wilbraham wrote:> Does your system have plenty of swap? I''ve been running a beefy dom0 > with at least 20 odd domUs running and dom0 has only had a fixed 512MB > allocated to it for the majority of its life.Host has 2GB of swap, so I doubt that''s the case here. However I managed to crash xend again and it seems to be somehow related to heavy disk I/O from domU. On this latest crash I had only two domU''s running and dom0 had plenty of memory to use. However I was running mkfs.ext3 on fairly large image which seemed to trigger the crash. First I lost an ssh connection to dom0 and regained it after a while, but xend was gone by then. I''ll try to debug the situation more, but if anyone has any ideas I''ll be glad to hear about them. After all my goal is (suprisingly) an stable system. One thing I did was that I increased dom0-min-memory even more, and it''s now at 512MB. Let''s see if that''s enough as Adam suggests or is the problem somewhere else. - -- Tapio Salonsaari My public GPG key is available at http://take.nerd.fi -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkov3O8ACgkQh/RMqfItS5Vk9wCePLDmeEBDRYKi42q5F1TanC/j 1PUAnjJm6RET4e8xjrjTYndYCz5X3VbO =nfUd -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tapio Salonsaari wrote:> I''ll try to debug the situation more, but if anyone has any ideas I''ll > be glad to hear about them. After all my goal is (suprisingly) an stableSorry to spam the list, but here''s some more information about what I discovered: # xm list Name ID Mem VCPUs State Domain-0 0 619 2 r----- LocalServer 1 128 1 -b---- pubservices 2 256 1 -b---- # head /proc/meminfo MemTotal: 633856 kB MemFree: 16704 kB Buffers: 2884 kB # cat /proc/swaps Filename Type Size Used /dev/sdc1 partition 979924 204 /dev/sdd1 partition 979924 0 There''s a little difference if LocalServer is running rsync at full speed or that the domU''s are idle. But that''s IMO quite logical that network traffic and disk I/O increases load on dom0 as well. However that consumption feels quite high, based on the feeling I got from reading this list, documentation etc. There''s no other services on dom0 besides the "normal" things (exim4, sshd etc). I started to suspect about the hardware itself as well, but sensors, hddtemp and others doesn''t show anything special and memory is ECC enabled and tested prior dom0 installation. That said, the system hasn''t crashed so far with dom0-min-mem 512 - -option, but the system has ran only an hour with that setting without any significant load, except the rsync -session. - -- Tapio Salonsaari My public GPG key is available at http://take.nerd.fi -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkov6+8ACgkQh/RMqfItS5WWxgCeIxnd3Y9/wDYLHqAxwZNgsXSq 4LUAoJd7TCwx1Otox9dzb9XFq+n2c8De =mOrO -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapio Salonsaari
2009-Jun-10 21:05 UTC
Re: [Xen-users] [SOLVED?] Xend stopped unexpectedly
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tapio Salonsaari wrote:> Sorry to spam the list, but here''s some more information about what I > discovered:Ok. It seems that I''ve found the solution with a little help from my friends. That''s most likely not about faulty hardware or lack of memory, but it''s a bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=524571 http://lists.xensource.com/archives/html/xen-users/2009-05/msg00389.html Obviously I should''ve be more descriptive with my question and tell you about the hardware and all. I promise I''ll be more spesific next time. - -- Tapio Salonsaari My public GPG key is available at http://take.nerd.fi -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkowIBUACgkQh/RMqfItS5VoKQCfSZzp6xSxH9zTFlRXv8HC/7Ew vekAniKalm70chwWNTSqU7dLOzG0CX37 =w6Tz -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users