Pkg xen devel - Nov 2014 - Bug#767295: [Xen-devel] [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel

On Thu, 2014-11-20 at 22:13 -0500, Gedalya wrote:
> On 11/20/2014 03:21 PM, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 20, 2014 at 03:48:47PM +0000, Ian Campbell wrote:
> >> The libxc xc_dom_* infrastructure uses a very simple malloc memory
pool which
> >> is freed by xc_dom_release. However the various xc_try_*_decode
routines (other
> >> than the gzip one) just use plain malloc/realloc and therefore the
buffer ends
> >> up leaked.
> >>
> >> The memory pool currently supports mmap'd buffers as well as a
directly
> >> allocated buffers, however the try decode routines make use of
realloc and do
> >> not fit well into this model. Introduce a concept of an external
memory block
> >> to the memory pool and provide an interface to register such
memory.
> >>
> >> The mmap_ptr and mmap_len fields of the memblock tracking struct
lose their
> >> mmap_ prefix since they are now also used for external memory
blocks.
> >>
> >> We are only seeing this now because the gzip decoder doesn't
leak and it's only
> >> relatively recently that kernels in the wild have switched to
better
> >> compression.
> >>
> >> This is https://bugs.debian.org/767295
> >>
> >> Reported by: Gedalya <gedalya at gedalya.net>
> > Gedelya,
> >
> > Could you also test this patch to make sure it does fix the
> > reported issue please?
> 
> So here's what happens now.
> 1. Starts up tiny
> 2. reboot: leak
> 3. reboot: freed (process larger, but the delta is all/mostly shared pages)
> 4. reboot: leak
> 5. reboot: freed
> etc..
WTF, how very strange!
> root at xen:~/xen-pkgs# xl cr /etc/xen/auto/asterisk_deb80.cfg
> Parsing config from /etc/xen/auto/asterisk_deb80.cfg
> root at xen:~/xen-pkgs# ps aux | grep asterisk_deb80
> root     22981  0.0  0.0  95968   588 ?        SLsl 21:55   0:00
/usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> root at xen:~/xen-pkgs# pmap -x 22981
> 22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> Address           Kbytes     RSS   Dirty Mode  Mapping
> 0000000000400000     144     128       0 r-x-- xl
> 0000000000623000       4       4       4 r---- xl
> 0000000000624000       8       8       8 rw--- xl
> 0000000000626000       4       4       4 rw---   [ anon ]
> 00000000009a6000     288     240     240 rw---   [ anon ]
> 00007f14d4000000     132       8       8 rw---   [ anon ]
> 00007f14d4021000   65404       0       0 -----   [ anon ]
> << snip >>
> ---------------- ------- ------- -------
> total kB           95968    2728     596
> 
> --- reboot domu ---
> 
> root at xen:~/xen-pkgs# ps aux | grep asterisk_deb80
> root     22981  0.6  3.3 131652 20008 ?        SLsl 21:55   0:00
/usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> root at xen:~/xen-pkgs# pmap -x 22981
> 22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> Address           Kbytes     RSS   Dirty Mode  Mapping
> 0000000000400000     144     144       0 r-x-- xl
> 0000000000623000       4       4       4 r---- xl
> 0000000000624000       8       8       8 rw--- xl
> 0000000000626000       4       4       4 rw---   [ anon ]
> 00000000009a6000     288     288     288 rw---   [ anon ]
> 00000000009ee000   35676   16772   16772 rw---   [ anon ]
This is the (temporarily) leaked mapping, right?
> Tried valgrind, it doesn't look like it was able to see what was going
on
Indeed. The values for total heap usage at exist and still reachable etc
also don't seem to account for the ~3M of mapping on each iteration.

I don't know how glibc's allocator works, but I suppose it isn't
impossible that it is retaining some mappings of free regions and
collecting them to free later somehow, which just happens to only
trigger every other reboot (e.g. perhaps it is based on some threshold
of free memory).

...investigates...

So, http://man7.org/linux/man-pages/man3/malloc.3.html talks about
special behaviour using mmap for allocations above MMAP_THRESHOLD (128K
by default), which we will be hitting here I think. That explains the
anon mapping.

http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
various dynamic thresholds for growing and shrinking the heap. My guess
is that we are bouncing up and down over some threshold with every other
reboot.

Ian.

On 11/21/2014 06:12 AM, Ian Campbell wrote:
> On Fri, 2014-11-21 at 11:03 +0000, Ian Campbell wrote:
>> http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
>> various dynamic thresholds for growing and shrinking the heap. My guess
>> is that we are bouncing up and down over some threshold with every
other
>> reboot.
> IOW I'm not overly concerned with this apparent bi-modality, so long as
> the amount isn't increasing in the long term...
>
> I think the original patch should go in.
>
> Ian.
>
>
It's an improvement, but consider this:
Someone has a xen host running wheezy, 40 domu's, with 768MB for dom0, 
worked fine so far. Tries upgrading to jessie, and lo, each domu process 
takes up only 588 KB on dom0, great!
Then a new kernel package is released, all domu's get rebooted once. All 
host memory is now full. Dude might have had other plans for that 
memory... This is dead memory so I guess it can be swapped out, not 
easily a scenario where the server totally crashes, but it's a bit ugly, 
we're talking about memory usage leaping from 0.6 to 16 MB per domu.