We''re running a handful of XCP 1.1 pools and have run in to situations wherein the local filesystem on the Dom0 control domain hits 100% capacity due to error logging in /var/log (daemon.log, messages, xensource.log, etc) thereby taking that host offline. Twice recently we''ve run in to this, on two different pools, first due to the primary SR hitting capacity and secondly when memory was exhausted on the host. My question is, how do others manage logs and such under these circumstances on Dom0? The first obvious answer seems to be tweaking /etc/logrotate.conf, but I was wondering if there are better methods seeing as this is XCP and not a more conventional Xen on Linux install. Thanks much. John Buchanan _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi John, On Thu, Jan 3, 2013 at 3:23 PM, John Buchanan <John.Buchanan@infinitecampus.com> wrote:> My question is, how do others manage logs and such under these circumstances > on Dom0? The first obvious answer seems to be tweaking /etc/logrotate.conf, > but I was wondering if there are better methods seeing as this is XCP and > not a more conventional Xen on Linux install.Monit is quite handy. You can use this to trigger a manual logrotate or call the logrotate binary. Of course you can also get it to alert you. http://mmonit.com/documentation/ Monit is very cross platform - it''d be a piece of cake to get it running under XCP. S.
Hi On Thu, Jan 3, 2013 at 3:23 PM, John Buchanan <John.Buchanan@infinitecampus.com> wrote:> the local filesystem on the Dom0 control domain hits 100% capacity due to > error logging in /var/log (daemon.log, messages, xensource.log, etc) thereby > taking that host offline.You can a) mount /var/log in a separate partition elsewhere to prevent the host from going offline; b) figure out why are those error logs full of, well, errors... and/or c) use whichever suggestions people throw in to deal with the logs. :) HTH, Nuno -- "On the internet, nobody knows you''re a dog."
Hello. El 03/01/13 09:23, John Buchanan escribió:> We''re running a handful of XCP 1.1 pools and have run in to situations > wherein the local filesystem on the Dom0 control domain hits 100% > capacity due to error logging in /var/log (daemon.log, messages, > xensource.log, etc) thereby taking that host offline. Twice recently > we''ve run in to this, on two different pools, first due to the primary > SR hitting capacity and secondly when memory was exhausted on the host.As a long term solution, I may suggest 2 ways to deal with excessive logging: 1. Adjust logrotate configuration. By default, it rotates the log files and deletes the oldest ones. Make it keep the logs for less time. 2. Store your logs on a remote server instead of local filesystem. XCP uses sysklogd, apparently it can send logs to a remote server in a natural way: http://www.cyberciti.biz/tips/log-all-logs-to-central-linux-unix-loghost.html> My question is, how do others manage logs and such under these > circumstances on Dom0? The first obvious answer seems to be tweaking > /etc/logrotate.conf, but I was wondering if there are better methods > seeing as this is XCP and not a more conventional Xen on Linux install.Both approaches i mentioned above has proven useful for me, individually and combined. I have not used this on XCP server (yet), but it works on a regular Debian with rsyslogd. -- Alexandre Kouznetsov
Nuno_Magalhães wrote:>You can a) mount /var/log in a separate partition elsewhere to prevent >the host from going offline;I'll second that suggestion. /var is the most likely cadidate for disk space problems on most system - so make sure it's a different filesystem to root. Do the same with any other areas (such as /home if users have file storage on it) as well. This goes a long way to dealing with the problem by avoiding the problems caused by a full filesystem - you will lose some logging, but it won't bring your system down in a horrible mess. And then try the other suggestions. Adjust whatever is creating the log entries, adjust logrotate (possibly run it more frequently than once a day), install something that will warn you if there's a problem (we have Nagios at work, though it's not the easiest thing to set up), and consider sending your logs elsewhere - though that doesn't really solve the problem of disk space, only move it. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi, 2013/1/3 John Buchanan <John.Buchanan@infinitecampus.com>:> We''re running a handful of XCP 1.1 pools and have run in to situations > wherein the local filesystem on the Dom0 control domain hits 100% capacity > due to error logging in /var/log (daemon.log, messages, xensource.log, etc) > thereby taking that host offline. Twice recently we''ve run in to this, on > two different pools, first due to the primary SR hitting capacity and > secondly when memory was exhausted on the host. > > My question is, how do others manage logs and such under these circumstances > on Dom0? The first obvious answer seems to be tweaking /etc/logrotate.conf, > but I was wondering if there are better methods seeing as this is XCP and > not a more conventional Xen on Linux install.Reducing useless messages is also a relevant thing to do - at least XenServer has an option to set the loglevel. I''d assume it''s in XCP too. And yes, monitoring is of course a necessity. ;) Better monitoring tools can monitor the grow rate over a monthly cycle and will as such only alert if logrotate fails it''s job. Greetings, Florian -- the purpose of libvirt is to provide an abstraction layer hiding all xen features added since 2006 until they were finally understood and copied by the kvm devs.
On Thu, 2013-01-03 at 15:23 +0000, John Buchanan wrote:> We''re running a handful of XCP 1.1 pools and have run in to situations > wherein the local filesystem on the Dom0 control domain hits 100% > capacity due to error logging in /var/log (daemon.log, messages, > xensource.log, etc) thereby taking that host offline.The XCP people tend to hang out on the (badly named) xen-api@ list so you might find you get answers there too. XCP 1.1 is a reasonably old release I think? I vaguely recall something about improvements to the amount of logging spew and/or rotation strategy in newer versions of XCP. Ian.