Hi! Suppose, there is a machine which writes two kinds of log files through syslogd: quickly-growing that need to be rotated based on their size (hourly is too seldom) and other that should be rotated once a day, at midnight only. For first kind of logs we have to run newsyslog once every 5 minutes using cron: */5 * * * * root newsyslog For second kind of logs we have lines in newsyslog.conf such as following: /var/log/mpd.log 640 16 * @T0000 JC This must ensure that /var/log/mpd.log is rotated and compressed at midnigt only. Note, that compressing the file takes 8 minutes. However, every night at 00:05 I get an error: bzip2: I/O or other error, bailing out. Possible reason follows. bzip2: No such file or directory Input file = /var/log/mpd.log.0, output file = /var/log/mpd.log.0.bz2 newsyslog: `bzip2 -f /var/log/mpd.log.0' terminated with a non-zero status (1) It seems, newsyslog still wants to process my file at 00:05 despite @T0000 time specification. Is it broken? Eugene Grosbein
31.07.2011 23:57, Freddie Cash ?????:> Simplest solution is to have two separate config files for newsyslogMy question is about possibly broken newsyslog, not how about "real task", it is described for clarification only. Eugene Grosbein
Simplest solution is to have two separate config files for newsyslog, and to edit /etc/crontab to point it at the two separate configs. One job runs every 5 mins, the other runs normally. I do this with three separate config files for different sets of logs that need to be rotated into different directories at different times. On Sun, Jul 31, 2011 at 9:51 AM, Eugene Grosbein <egrosbein@rdtc.ru> wrote:> Hi! > > Suppose, there is a machine which writes two kinds of log files through > syslogd: > quickly-growing that need to be rotated based on their size (hourly is too > seldom) > and other that should be rotated once a day, at midnight only. > > For first kind of logs we have to run newsyslog once every 5 minutes using > cron: > > */5 * * * * root newsyslog > > For second kind of logs we have lines in newsyslog.conf such as following: > > /var/log/mpd.log 640 16 * @T0000 JC > > This must ensure that /var/log/mpd.log is rotated and compressed at midnigt > only. > Note, that compressing the file takes 8 minutes. > > However, every night at 00:05 I get an error: > > bzip2: I/O or other error, bailing out. Possible reason follows. > bzip2: No such file or directory > Input file = /var/log/mpd.log.0, output file > /var/log/mpd.log.0.bz2 > newsyslog: `bzip2 -f /var/log/mpd.log.0' terminated with a non-zero status > (1) > > It seems, newsyslog still wants to process my file at 00:05 despite @T0000 > time specification. Is it broken? > > Eugene Grosbein > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >-- Freddie Cash fjwcash@gmail.com
On Sun, Jul 31, 2011 at 11:51:40PM +0700, Eugene Grosbein wrote:> Hi! > > Suppose, there is a machine which writes two kinds of log files through syslogd: > quickly-growing that need to be rotated based on their size (hourly is too seldom) > and other that should be rotated once a day, at midnight only. > > For first kind of logs we have to run newsyslog once every 5 minutes using cron: > > */5 * * * * root newsyslog > > For second kind of logs we have lines in newsyslog.conf such as following: > > /var/log/mpd.log 640 16 * @T0000 JC > > This must ensure that /var/log/mpd.log is rotated and compressed at midnigt only. > Note, that compressing the file takes 8 minutes. > > However, every night at 00:05 I get an error: > > bzip2: I/O or other error, bailing out. Possible reason follows. > bzip2: No such file or directory > Input file = /var/log/mpd.log.0, output file = /var/log/mpd.log.0.bz2 > newsyslog: `bzip2 -f /var/log/mpd.log.0' terminated with a non-zero status (1) > > It seems, newsyslog still wants to process my file at 00:05 despite @T0000 > time specification. Is it broken?I have three things to say on the matter, all of which are somewhat independent of one another so please keep that in mind. I imagine #1 below is your problem. 1) The newsyslog.conf(5) man page has this clause in it, for the "when" field (in your case, @T0000): when ... If the when field contains an asterisk (`*'), log rotation will solely depend on the contents of the size field. Otherwise, the when field consists of an optional interval in hours, usually followed by an `@'-sign and a time in restricted ISO 8601 format. If a time is specified, the log file will only be trimmed if newsyslog(8) is run within one hour of the specified time. If an interval is specified, the log file will be trimmed if that many hours have passed since the last rotation. ... You might think that "one hour of the specified time" value/clause correlates with the interval that newsyslog is run at via cron, but that would be wrong. newsyslog REALLY DOES have hard-coded values for 3600 seconds (1 hour) in it (grep -r 3600 /usr/src/usr.sbin/newsyslog). I have not looked at the code, but the fact of the matter is, 1 hour appears to be a "special" value. I would heed that as a warning. 2) Are you absolutely sure mpd.log is being rotated AND compressed within the 5 minute window? If mpd.log is extremely large and your disks are slow, this could take a long time. If possible, try (temporarily) removing bzip2 from the picture (remove J flag). 3) mpd(8) logs via syslog(3). When newsyslog(8), are you aware that it sends a SIGHUP to syslogd(8)? As such, are you absolutely certain when this happen (every 5 minutes!) that the new log files are getting created correctly and promptly? 4) To debug this, you're probably going to need to run some cronjobs or daemons that keep a very close eye on /var/log/mpd.log* when the log rotation runs, in combination with running syslogd(8) in debug mode and/or verbose mode. 5) Why do you need to rotate logs every 5 minutes? Why do you need such extreme levels of granularity in your rotated logs? Just how much data are you logging via syslog? If a lot, why so much? It might be more effective to consider expanding your logging infrastructure to multiple machines if this the case. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
01.08.2011 00:31, Jeremy Chadwick ?????:> On Sun, Jul 31, 2011 at 11:51:40PM +0700, Eugene Grosbein wrote: >> Hi! >> >> Suppose, there is a machine which writes two kinds of log files through syslogd: >> quickly-growing that need to be rotated based on their size (hourly is too seldom) >> and other that should be rotated once a day, at midnight only. >> >> For first kind of logs we have to run newsyslog once every 5 minutes using cron: >> >> */5 * * * * root newsyslog >> >> For second kind of logs we have lines in newsyslog.conf such as following: >> >> /var/log/mpd.log 640 16 * @T0000 JC >> >> This must ensure that /var/log/mpd.log is rotated and compressed at midnigt only. >> Note, that compressing the file takes 8 minutes. >> >> However, every night at 00:05 I get an error: >> >> bzip2: I/O or other error, bailing out. Possible reason follows. >> bzip2: No such file or directory >> Input file = /var/log/mpd.log.0, output file = /var/log/mpd.log.0.bz2 >> newsyslog: `bzip2 -f /var/log/mpd.log.0' terminated with a non-zero status (1) >> >> It seems, newsyslog still wants to process my file at 00:05 despite @T0000 >> time specification. Is it broken? > > I have three things to say on the matter, all of which are somewhat > independent of one another so please keep that in mind. I imagine #1 > below is your problem. > > 1) The newsyslog.conf(5) man page has this clause in it, for the "when" > field (in your case, @T0000): > > when ... If the when field contains an asterisk (`*'), log rotation > will solely depend on the contents of the size field. Otherwise, > the when field consists of an optional interval in hours, usually > followed by an `@'-sign and a time in restricted ISO 8601 format. > > If a time is specified, the log file will only be trimmed if > newsyslog(8) is run within one hour of the specified time. If an > interval is specified, the log file will be trimmed if that many > hours have passed since the last rotation. ... > > You might think that "one hour of the specified time" value/clause > correlates with the interval that newsyslog is run at via cron, but that > would be wrong. newsyslog REALLY DOES have hard-coded values for 3600 > seconds (1 hour) in it (grep -r 3600 /usr/src/usr.sbin/newsyslog). I > have not looked at the code, but the fact of the matter is, 1 hour > appears to be a "special" value. I would heed that as a warning.After reading newsyslog code, now it's obvious it just ignores minutes and seconds while making decision if a file should be rotated. It looks at hours only. That's sad. Eugene Grosbein