On Wed, May 04, 2011 at 02:58:20PM -0400, Mikhail T.
wrote:> On FreeBSD/amd64 8.2-STABLE as of Sun Feb 27. I dismissed this
> problem the first time (a week or two ago), but just saw it again.
> Something triggers syslogd into spinning all available CPU -- while
> not logging anything.
>
> Attempts to log anything -- such as by invoking logger(1) -- simply
> hang. sendmail hangs too -- now new e-mail is coming. It is not
> pretty.
>
> ktrace-ing the process yields empty file -- it is not doing
> system-calls, while spinning.
>
> Attaching the debugger several times shows the same stack:
>
> #0 0x00000008007cb1d6 in _pthread_mutex_init_calloc_cb () from
/lib/libc.so.7
> #1 0x00000008007cd9fb in _pthread_mutex_init_calloc_cb () from
/lib/libc.so.7
> #2 0x00000008007d4825 in free () from /lib/libc.so.7
> #3 0x00000008007c5aab in catopen () from /lib/libc.so.7
> #4 0x00000008007c5410 in strerror_r () from /lib/libc.so.7
> #5 0x00000008007c556d in strerror () from /lib/libc.so.7
> #6 0x0000000000403c10 in ?? ()
> ...
>
> Among the logging-destinations in my syslog.conf there is a program,
> that exits sometimes -- but never "too fast". Normally, syslogd
> simply restarts it without an issue...
>
> Any ideas? Thanks!
If ktrace doesn't show any syscalls (did you use "ktrace -i
-t+"?), then
I'm not sure how the process is actually spinning (100% CPU).
"procstat
-kk" on the PID of syslogd might be more useful in this case. The
output is mainly useful for kernel folks. You may end up having to drop
to ddb to examine what's going on within the pid.
I'm not sure if attaching gdb to a running process on FreeBSD works with
things like threads, etc.. Meaning I'm not sure if the above stack
trace is indicative of a problem or not. No offence to anyone intended,
but I try to avoid gdb on FreeBSD wherever possible (it's so old that I
don't know how useful it is for extreme debugging).
I've never seen syslogd spiral out of control on any of our systems, but
all logging destinations in syslog.conf on our systems are files on
local disks.
If you can find a way to reliably reproduce the issue that would be
useful, otherwise this sounds like one of those things that's going to
be a problem to track down. :-(
--
| Jeremy Chadwick jdc@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP 4BD6C0CB |