thr3ads.net - CentOS - [CentOS] nfsen daemon dies for unknown reason [Apr 2012]

If this information is useful, please help other people find it:
Share via:

Nikolaos Milas

2012-Apr-21 22:14 UTC

[CentOS] nfsen daemon dies for unknown reason

Hello,

I am using nfsen 1.3.5 (http://nfsen.sourceforge.net/) on CentOS 5.8 x86_64
(running as a VM on a KVM Host).

For those not knowing it, nfsen is using php and perl and it allows graphing and
analysis of (Cisco routers) netflow data captured using nfdump.

Often (about once a day) nfsen crashes for unknown reason. For example:

     # /data/nfsen/bin/nfsen status
     NfSen version: 1.3.5
     NfSen status:
     Collector for (pen thi) port 9995 is running [2688].
     nfsen daemon:  pid: [2690] died for unknown reason.

We are running nfsen by including in /etc/rc.local:

     /data/nfsen/bin/nfsen start

Note that nfdump 1.6.4 (used by nfsen) is running flawlessly, never crashing.
When restarting:

     # /data/nfsen/bin/nfsen start
     Starting nfcapd:(pen thi)
     Error( Port: 9995 ): a collector with pid[2688] is already running

     Starting nfsend.

nfsen and nfdump were compiled from source.

Note that I haven't had such issues with any other software, except nc
(netcat) which would also hang in the same way. After many troubles and
unsuccessful efforts to resolve, I was forced to use socat instead; socat has
been running flawlessly for months. (This might be a different issue though.)

Could someone please try to identify the source of these problems and suggest a
solution?

I've tried to find more details on the problem (see below). I tried to
collect some debug info (I'm not an expert) and in the log I see the
following about the first failed run (after about twelve hours):

-------------------------------------------------------------------------------------

Apr 20 21:30:29 monserver nfsen[6933]: connection on UNIX socket
Apr 20 21:30:29 monserver nfsen[6933]: comm server started: 18240
Apr 20 21:30:39 monserver nfsen[18240]: Failed: Timeout reading from socket! at
/data/nfsen/libexec/Nfcomm.pm line 1131.
Apr 20 21:30:39 monserver nfsen[6933]: comm child[18240] terminated with no exit
value

-------------------------------------------------------------------------------------

When running:

(It seems that the multiple nfsend-comm processes remain from previous runs of
nfsen; when nfsen dies "for unknown reason", I run nfsen start and
nfsen starts but an nfsend-comm from the previous run remains in the process
list.)

# ps axjf
  PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
...
     1 30577 30577 30577 ?           -1 Ss     504   0:07
/data/nfsen/bin/nfsend-comm
     1 21259 21259 21259 ?           -1 Ss     504   0:01
/data/nfsen/bin/nfsend-comm
     1 11816 11816 11816 ?           -1 Ss     504   0:00
/data/nfsen/bin/nfsend-comm
     1 19817 19817 19817 ?           -1 Ss     504   0:01
/data/nfsen/bin/nfsend-comm
     1 25822 25822 25822 ?           -1 Ss     504   0:04
/data/nfsen/bin/nfsend-comm
     1  2131  2131  2131 ?           -1 Ss     504   0:03
/data/nfsen/bin/nfsend-comm
...
     1  6932  6932  6932 ?           -1 Rs     504   2:40 /usr/bin/perl -w
/data/nfsen/bin/nfsend
  6932  6933  6933  6933 ?           -1 Ss     504   0:00  \_
/data/nfsen/bin/nfsend-comm


# lsof -p 6932
COMMAND  PID  USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
nfsend  6932 nfsen  cwd    DIR              252,0     4096        2 /
nfsend  6932 nfsen  rtd    DIR              252,0     4096        2 /
nfsend  6932 nfsen  txt    REG              252,0    19200    75700
/usr/bin/perl
nfsend  6932 nfsen  mem    REG              252,0   144776  1301763
/lib64/ld-2.5.so
nfsend  6932 nfsen  mem    REG              252,0  1722328  1301781
/lib64/libc-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    23360  1301798
/lib64/libdl-2.5.so
nfsend  6932 nfsen  mem    REG              252,0   615136  1301995
/lib64/libm-2.5.so
nfsend  6932 nfsen  mem    REG              252,0   145872  1301799
/lib64/libpthread-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    85544  1301997
/lib64/libz.so.1.2.3
nfsend  6932 nfsen  mem    REG              252,0    53448  1301800
/lib64/librt-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    51280    73941
/usr/lib64/libdbi.so.0.0.5
nfsend  6932 nfsen  mem    REG              252,0   690576    78964
/usr/lib64/librrd.so.4.1.4
nfsend  6932 nfsen  mem    REG              252,0  1262480   228801
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
nfsend  6932 nfsen  mem    REG              252,0    12040    72516
/usr/lib64/libXau.so.6.0.0
nfsend  6932 nfsen  mem    REG              252,0    22032    72673
/usr/lib64/libXdmcp.so.6.0.0
nfsend  6932 nfsen  mem    REG              252,0  1099816    72985
/usr/lib64/libX11.so.6.2.0
nfsend  6932 nfsen  mem    REG              252,0   262904  1301946
/lib64/libgobject-2.0.so.0.1200.3
nfsend  6932 nfsen  mem    REG              252,0   114352  1302051
/lib64/libnsl-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    13520  1301945
/lib64/libgmodule-2.0.so.0.1200.3
nfsend  6932 nfsen  mem    REG              252,0    92816  1302056
/lib64/libresolv-2.5.so
nfsend  6932 nfsen  mem    REG              252,0   143144  1302011
/lib64/libexpat.so.0.5.0
nfsend  6932 nfsen  mem    REG              252,0   543824    69252
/usr/lib64/libfreetype.so.6.3.10
nfsend  6932 nfsen  mem    REG              252,0    38528    73002
/usr/lib64/libXrender.so.1.3.0
nfsend  6932 nfsen  mem    REG              252,0   212896    69259
/usr/lib64/libfontconfig.so.1.1.0
nfsend  6932 nfsen  mem    REG              252,0   432992    73825
/usr/lib64/libcairo.so.2.9.2
nfsend  6932 nfsen  mem    REG              252,0   149344    70222
/usr/lib64/libpng12.so.0.10.0
nfsend  6932 nfsen  mem    REG              252,0  1297136    76276
/usr/lib64/libxml2.so.2.6.26
nfsend  6932 nfsen  mem    REG              252,0    39304    74098
/usr/lib64/libpangocairo-1.0.so.0.1400.9
nfsend  6932 nfsen  mem    REG              252,0   192152    73833
/usr/lib64/libpangoft2-1.0.so.0.1400.9
nfsend  6932 nfsen  mem    REG              252,0   265536    69216
/usr/lib64/libpango-1.0.so.0.1400.9
nfsend  6932 nfsen  mem    REG              252,0    18152  1302075
/lib64/libutil-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    48600  1302080
/lib64/libcrypt-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    25920  2701809
/var/cache/fontconfig/beeeeb3dfe132a8a0633a017c99ce0c0-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0 56414128    69193
/usr/lib/locale/locale-archive
nfsend  6932 nfsen  mem    REG              252,0   119224   229644
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
nfsend  6932 nfsen  mem    REG              252,0    21424   229774
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
nfsend  6932 nfsen  mem    REG              252,0    14840   229057
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
nfsend  6932 nfsen  mem    REG              252,0    15880   229770
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so
nfsend  6932 nfsen  mem    REG              252,0    31656   229457
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
nfsend  6932 nfsen  mem    REG              252,0    10800   228792
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
nfsend  6932 nfsen  mem    REG              252,0    37400   229771
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
nfsend  6932 nfsen  mem    REG              252,0    97334   619478
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so
nfsend  6932 nfsen  mem    REG              252,0    44016  2701801
/var/cache/fontconfig/e19de935dec46bbf3ed114ee4965548a-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    10136  2701251
/var/cache/fontconfig/e3ead4b767b8819993a6fa3ae306afa9-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0   647608  1301942
/lib64/libglib-2.0.so.0.1200.3
nfsend  6932 nfsen  mem    REG              252,0    13056   228836
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
nfsend  6932 nfsen  mem    REG              252,0    81800   229709
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
nfsend  6932 nfsen  mem    REG              252,0    18080   228851
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
nfsend  6932 nfsen  mem    REG              252,0    28544  2701803
/var/cache/fontconfig/7ddba6133ef499da58de5e8c586d3b75-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    53880  1301825
/lib64/libnss_files-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    66360  2701404
/var/cache/fontconfig/73a61b34dd8ca4d8a159807604ab432f-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    32184  2701806
/var/cache/fontconfig/b79f3aaa7d385a141ab53ec885cc22a8-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0     7184  2701804
/var/cache/fontconfig/87f5e051180a7a75f16eb6fe7dbd3749-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    11704   358189
/usr/lib64/pango/1.5.0/modules/pango-basic-fc.so
nfsend  6932 nfsen  mem    REG              252,0   237540   618666
/usr/share/fonts/dejavu-lgc/DejaVuLGCSansMono.ttf
nfsend  6932 nfsen    0r   CHR                1,3      0t0     1204 /dev/null
nfsend  6932 nfsen    1w   CHR                1,3      0t0     1204 /dev/null
nfsend  6932 nfsen    2w   CHR                1,3      0t0     1204 /dev/null
nfsend  6932 nfsen    3u  unix 0xffff8100a7cadc40      0t0 10902452 socket

-------------------------------------------------------------------------------------

I also tried to run gdb at the same time:

# gdb /data/nfsen/bin/nfsend 6932
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or
later<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
"/data/nfsen/bin/nfsend": not in executable format: File format not
recognized
Attaching to process 6932
Reading symbols from /usr/bin/perl...(no debugging symbols found)...done.
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...(no debugging
symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so...(no debugging
symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
Reading symbols from
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so...done.
Loaded symbols for
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so
Reading symbols from /usr/lib64/librrd.so.4...done.
Loaded symbols for /usr/lib64/librrd.so.4
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libdbi.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libdbi.so.0
Reading symbols from /usr/lib64/libpng12.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpng12.so.0
Reading symbols from /usr/lib64/libpangocairo-1.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpangocairo-1.0.so.0
Reading symbols from /usr/lib64/libpango-1.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpango-1.0.so.0
Reading symbols from /usr/lib64/libcairo.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libcairo.so.2
Reading symbols from /lib64/libgobject-2.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgobject-2.0.so.0
Reading symbols from /lib64/libgmodule-2.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgmodule-2.0.so.0
Reading symbols from /lib64/libglib-2.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libglib-2.0.so.0
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libpangoft2-1.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpangoft2-1.0.so.0
Reading symbols from /usr/lib64/libfontconfig.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libfontconfig.so.1
Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libXrender.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libXrender.so.1
Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libX11.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libexpat.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libexpat.so.0
Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libXau.so.6
Reading symbols from /usr/lib64/libXdmcp.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libXdmcp.so.6
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so...(no debugging
symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so

warning: no loadable sections found in added symbol-file system-supplied DSO at
0x7fff95ffd000
0x000000372a09a8d0 in __nanosleep_nocancel () from /lib64/libc.so.6
(gdb) continue
Continuing.
Detaching after fork from child process 7558.
Detaching after fork from child process 7559.
...
<Many identical messages: Detaching after fork from child process xxxxx.>
...
Detaching after fork from child process 18133.
Detaching after fork from child process 18135.

Program received signal SIGPIPE, Broken pipe.
0x000000372ac0df15 in send () from /lib64/libpthread.so.0

(gdb) backtrace full
#0  0x000000372ac0df15 in send () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x000000372c0cf063 in Perl_pp_send () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#2  0x000000372c08a39e in Perl_runops_standard () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#3  0x000000372c0380ba in perl_run () from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#4  0x00000000004017bc in main ()
No symbol table info available.

-------------------------------------------------------------------------------------

Does that help at all?

If you would like to guide me to further debug, please provide details on what
should I do. (e.g. how to install debug symbols for libperl.so and
libpthread.so)

Please advise.

Thanks,
Nick

Markus Falb

2012-Apr-22 10:28 UTC

head link

[CentOS] nfsen daemon dies for unknown reason

On 22.4.2012 00:14, Nikolaos Milas wrote:
> I am using nfsen 1.3.5 (http://nfsen.sourceforge.net/) on CentOS 5.8 x86_64
(running as a VM on a KVM Host).
> 
> For those not knowing it, nfsen is using php and perl and it allows
graphing and analysis of (Cisco routers) netflow data captured using nfdump.
> 
> Often (about once a day) nfsen crashes for unknown reason. For example:
...> please provide details on what should I do. (e.g. how to install debug
symbols for libperl.so and libpthread.so)
At least it can't hurt to install the debug symbols ;-)
debuginfo packages are at http://debuginfo.centos.org

...snip
$ yum --enablerepo=debug list glibc-debuginfo perl-debuginfo
Available Packages
glibc-debuginfo.x86_64               2.5-81.el5_8.1                debug
perl-debuginfo.x86_64                4:5.8.8-38.el5                debug
snap...

It seems that the debuginfo package for the latest glibc update
(glibc-2.5-81.el5_8.2) is missing
-- 
Kind Regards, Markus Falb

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 307 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20120422/8f18bf86/attachment-0002.sig>

Nikolaos Milas

2012-May-08 11:41 UTC

head link

[CentOS] nfsen daemon dies for unknown reason

On 22/4/2012 1:28 ??, Markus Falb wrote:
>> I am using nfsen 1.3.5 (http://nfsen.sourceforge.net/) on CentOS 5.8
x86_64 (running as a VM on a KVM Host).
>> >  
>> >  For those not knowing it, nfsen is using php and perl and it
allows graphing and analysis of (Cisco routers) netflow data captured using
nfdump.
>> >  
>> >  Often (about once a day) nfsen crashes for unknown reason. For
example:
> ...
>
>> >  please provide details on what should I do. (e.g. how to install
debug symbols for libperl.so and libpthread.so)
>>
> At least it can't hurt to install the debug symbols
Thanks Markus,

I installed the debugging symbols, and below follows a backtrace from a 
hang.

Can somebody please help in resolving the issue? Does this backtrace help?

Any other advice to resolve the issue with hangs?

Thanks,
Nick

============================================
# gdb /data/nfsen/bin/nfsend 13540
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
"/data/nfsen/bin/nfsend": not in executable format: File format not 
recognized
Attaching to process 13540
...
/Reading symbols...
...
/ (gdb) continue
Continuing.
/Detaching after fork from child process xxxxx.
Detaching after fork from child process yyyyy.
Detaching after fork from child process zzzzz./
...
Program received signal SIGPIPE, Broken pipe.
0x00000036e040df15 in __libc_send (fd=4, buf=0x473f250, n=12, flags=0) 
at ../sysdeps/unix/sysv/linux/x86_64/send.c:28
28 return INLINE_SYSCALL (sendto, 6, fd, buf, n, flags, NULL, NULL);
(gdb) backtrace full
#0 0x00000036e040df15 in __libc_send (fd=4, buf=0x473f250, n=12, 
flags=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:28
oldtype = <value optimized out>
result = <value optimized out>
#1 0x00000036e18cf063 in Perl_pp_send (my_perl=0x3bc4010) at pp_sys.c:1934
sp = <value optimized out>
mark = 0x3caa498
origmark = 0
targ = 0x468c000
gv = 0x4717f30
io = 0x4727fe0
bufsv = 0x468bfd0
buffer = 0x473f250 ".timeout=10\n"
length = <value optimized out>
retval = <value optimized out>
blen = 12
mg = <value optimized out>
#2 0x00000036e188a39e in Perl_runops_standard (my_perl=0x3bc4010) at 
run.c:37
No locals.
#3 0x00000036e18380ba in S_run_body (my_perl=0x3bc4010) at perl.c:2372
No locals.
#4 perl_run (my_perl=0x3bc4010) at perl.c:2289
oldscope = 1
ret = <value optimized out>
cur_env = {je_prev = 0x3bc42a8, je_buf = {{__jmpbuf = {140733567184264, 
8312953106376043799, 0, 140733567184256, 0, 0, 8313070080049120900,
8312953287326650575}, __mask_was_saved = 0, __saved_mask = {__val = {11, 
235712106880, 0, 0, 0, 62673248, 62668816, 62668816, 140733567184256,
62673248, 62668816, 235712042554, 0, 62669480, 0, 140733567184264}}}}, 
je_ret = 0, je_mustcatch = 0 '\000'}
#5 0x00000000004017bc in main (argc=3, argv=0x7fff1647ad88, env=<value 
optimized out>) at perlmain.c:99
exitstatus = -32

==========================================================

Possibly Parallel Threads

Search for more maybe matching threads

CentOS - Apr 2012 - nfsen daemon dies for unknown reason

[CentOS] nfsen daemon dies for unknown reason

[CentOS] nfsen daemon dies for unknown reason

[CentOS] nfsen daemon dies for unknown reason

Possibly Parallel Threads