Libguestfs - Aug 2020 - Long running nbdkit instances seem to leak memory

I'm using nbdkit + the file plugin to serve NBD root filesystems for
some machines, so I get to observe how it behaves for very long runs.
I think this indicates a memory leak?

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 5038 root      20   0 4577220  16000    796 S   0.0  0.1  30:52.40 nbdkit
 5592 root      20   0 4159224  11708    784 S   0.0  0.1  22:21.67 nbdkit

Not sure how to easily diagnose this.  My thought was to force a
coredump by sending a signal, but I don't know if these processes have
a suitable rlimit.  I could also attach gdb.  Perhaps by looking at a
dump of the memory we could get an idea of what structure leaks.

Unfortunately also it's a very old version:

$ rpm -qf /usr/sbin/nbdkit 
nbdkit-1.8.0-1.el7.x86_64

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

May I ask for self-education, how did you conclude memory leak, based on
that, I guess, 'ps' output?

--
  Mykola Ivanets

пн, 10 серп. 2020, 10:48 користувач Richard W.M. Jones <rjones@redhat.com>
пише:
> I'm using nbdkit + the file plugin to serve NBD root filesystems for
> some machines, so I get to observe how it behaves for very long runs.
> I think this indicates a memory leak?
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>  5038 root      20   0 4577220  16000    796 S   0.0  0.1  30:52.40 nbdkit
>  5592 root      20   0 4159224  11708    784 S   0.0  0.1  22:21.67 nbdkit
>
> Not sure how to easily diagnose this.  My thought was to force a
> coredump by sending a signal, but I don't know if these processes have
> a suitable rlimit.  I could also attach gdb.  Perhaps by looking at a
> dump of the memory we could get an idea of what structure leaks.
>
> Unfortunately also it's a very old version:
>
> $ rpm -qf /usr/sbin/nbdkit
> nbdkit-1.8.0-1.el7.x86_64
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines.  Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
>
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs
>
>

On Mon, Aug 10, 2020 at 10:56:58AM +0300, Nikolay Ivanets
wrote:
> May I ask for self-education, how did you conclude memory leak, based on
> that, I guess, 'ps' output?
ps output isn't very informative, although it shows a large virtual
size.  The real information came from looking at /proc/PID/maps and
/proc/PID/smaps which show very large numbers of 8MB anonymous
segments, which could be caused by malloc never being freed (or just
fragmentation - I'm not clear right now).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org