Icecast - Oct 2020 - Icecast crashing / terminated - out of memory

Hi, needing some advice. 
I am running a backports version of icecast for Debian 10. I have found that
with this version, the icecast server will usually run for about 7 days before
it crashes. I found in the syslog that icecast gets terminated because it?s
'out of memory'.

This never happened on the standard (stable) version of icecast that I ran on
Debian 9 installed via apt.
I?ve added the relevant section of the syslog below. Any help on steps to take
would be greatly appreciated.


Oct 10 10:36:40 systrum kernel: [925937.080802] 0 pages HighMem/MovableOnly
Oct 10 10:36:40 systrum kernel: [925937.082050] 53826 pages reserved
Oct 10 10:36:40 systrum kernel: [925937.083128] Tasks state (memory values in
pages):
Oct 10 10:36:40 systrum kernel: [925937.084663] [  pid  ]   uid  tgid total_vm  
rss pgtables_bytes swapents oom_score_adj name
Oct 10 10:36:40 systrum kernel: [925937.087367] [    227]     0   227    11670  
1758   110592        0             0 systemd-journal
Oct 10 10:36:40 systrum kernel: [925937.090230] [    240]     0   240     5018  
228    65536        0         -1000 systemd-udevd
Oct 10 10:36:40 systrum kernel: [925937.093264] [    312]     0   312     2110  
70    53248        0             0 cron
Oct 10 10:36:40 systrum kernel: [925937.096122] [    313]     0   313    59770  
289    98304        0             0 polkitd
Oct 10 10:36:40 systrum kernel: [925937.098769] [    314]     0   314    56433  
391    77824        0             0 rsyslogd
Oct 10 10:36:40 systrum kernel: [925937.111310] [    315]   104   315     2220  
198    57344        0          -900 dbus-daemon
Oct 10 10:36:40 systrum kernel: [925937.114461] [    316]   105   316      629  
33    40960        0             0 nscd
Oct 10 10:36:40 systrum kernel: [925937.117285] [    318]     0   318     4849  
256    73728        0             0 systemd-logind
Oct 10 10:36:40 systrum kernel: [925937.120289] [    319]     0   319     1608  
56    49152        0             0 qemu-ga
Oct 10 10:36:40 systrum kernel: [925937.123219] [    400]     0   400     2372  
326    53248        0             0 dhclient
Oct 10 10:36:40 systrum kernel: [925937.125971] [    489]     0   489     2372  
324    53248        0             0 dhclient
Oct 10 10:36:40 systrum kernel: [925937.128553] [    592]     0   592     2372  
322    57344        0             0 dhclient
Oct 10 10:36:40 systrum kernel: [925937.131123] [    663]     0   663    27551  
1912   114688        0             0 unattended-upgr
Oct 10 10:36:40 systrum kernel: [925937.133978] [    665]     0   665    57651  
2386   196608        0             0 php-fpm7.3
Oct 10 10:36:40 systrum kernel: [925937.136604] [    669]     0   669    65100  
2885   135168        0             0 tuned
Oct 10 10:36:40 systrum kernel: [925937.139112] [    686]     0   686     1316  
16    49152        0             0 agetty
Oct 10 10:36:40 systrum kernel: [925937.141673] [    689]     0   689      641  
18    40960        0             0 agetty
Oct 10 10:36:40 systrum kernel: [925937.144260] [    699]   106   699    19117  
146    61440        0             0 ntpd
Oct 10 10:36:40 systrum kernel: [925937.146907] [    720]    33   720    57651  
1770   176128        0             0 php-fpm7.3
Oct 10 10:36:40 systrum kernel: [925937.150091] [    721]    33   721    57651  
1770   176128        0             0 php-fpm7.3
Oct 10 10:36:40 systrum kernel: [925937.153211] [   1119]     0  1119    10869  
211    81920        0             0 master
Oct 10 10:36:40 systrum kernel: [925937.156248] [   1123]   110  1123    10892  
231    81920        0             0 qmgr
Oct 10 10:36:40 systrum kernel: [925937.159217] [  22760]     0 22760     3942  
214    69632        0         -1000 sshd
Oct 10 10:36:40 systrum kernel: [925937.162157] [  28701]     0 28701   304811  
4989   262144        0             0 fail2ban-server
Oct 10 10:36:40 systrum kernel: [925937.165398] [   8095]   108  8095  1988579 
1896082 15364096        0             0 icecast2
Oct 10 10:36:40 systrum kernel: [925937.168408] [   8240]  1000  8240   152698  
21447   344064        0             0 liquidsoap
Oct 10 10:36:40 systrum kernel: [925937.171386] [   8432]  1000  8432   151856  
20770   331776        0             0 liquidsoap
Oct 10 10:36:40 systrum kernel: [925937.174400] [  17839]     0 17839    58512  
2828   204800        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.177286] [  30783]    33 30783    77224  
3124   217088        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.180337] [  10477]    33 10477    77171  
3087   212992        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.183216] [  13167]    33 13167    77184  
3098   217088        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.186299] [  18536]    33 18536    77172  
3087   217088        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.189379] [  21245]    33 21245    77164  
3091   212992        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.192333] [  24426]    33 24426    77195  
3078   217088        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.195363] [  24885]    33 24885    77169  
3079   212992        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.198359] [  25783]    33 25783    77180  
3077   217088        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.201343] [  28454]    33 28454    77169  
3087   217088        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.204308] [  29382]    33 29382    77178  
3080   212992        0             0 apache2
Oct 10 10:36:40 systrum kernel: [925937.207252] [   2631]   110  2631    10852  
198    81920        0             0 pickup
Oct 10 10:36:40 systrum kernel: [925937.210233] Out of memory: Kill process 8095
(icecast2) score 929 or sacrifice child
Oct 10 10:36:40 systrum kernel: [925937.213229] Killed process 8095 (icecast2)
total-vm:7954316kB, anon-rss:7584328kB, file-rss:0kB, shmem-rss:0kB

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20201010/be2c5e5d/attachment.htm>

Good morning,

first of all, thank you for your nice report. :)

On Sat, 2020-10-10 at 20:32 +1000, Damian wrote:
> Hi, needing some advice.
> I am running a backports version of icecast for Debian 10. I have
> found that with this version, the icecast server will usually run for
> about 7 days before it crashes. I found in the syslog that icecast
> gets terminated because it?s 'out of memory'.
There is currently a problem known in wolfssl which leads to Icecast
leaking memory. See [0], [1].
> This never happened on the standard (stable) version of icecast that I
> ran on Debian 9 installed via apt.
> I?ve added the relevant section of the syslog below. Any help on steps
> to take would be greatly appreciated.
That matches the picture.

Do you have TLS enabled?

Please provide a: cat /proc/$(pidof icecast2)/maps

This allows us to verify if it's the known problem. You can run the
command directly after you started Icecast.


With best regards,
> 
> Oct 10 10:36:40 systrum kernel: [925937.080802] 0 pages HighMem/MovableOnly
> Oct 10 10:36:40 systrum kernel: [925937.082050] 53826 pages reserved
> Oct 10 10:36:40 systrum kernel: [925937.083128] Tasks state (memory values
in pages):
> Oct 10 10:36:40 systrum kernel: [925937.084663] [  pid  ]   uid  tgid
total_vm      rss pgtables_bytes swapents oom_score_adj name
> [...]
> Oct 10 10:36:40 systrum kernel: [925937.165398] [   8095]   108  8095 
1988579  1896082 15364096        0             0 icecast2
> [...]
> Oct 10 10:36:40 systrum kernel: [925937.210233] Out of memory: Kill process
8095 (icecast2) score 929 or sacrifice child
> Oct 10 10:36:40 systrum kernel: [925937.213229] Killed process 8095
(icecast2) total-vm:7954316kB, anon-rss:7584328kB, file-rss:0kB, shmem-rss:0kB
> 
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969018
[1] https://gitlab.xiph.org/xiph/icecast-server/-/issues/2391


-- 
Philipp Schafft (CEO/Gesch?ftsf?hrer) 
Telephon: +49.3535 490 17 92

L?wenfelsen UG (haftungsbeschr?nkt)     Registration number:
Bickinger Stra?e 21                     HRB 12308 CB
04916 Herzberg (Elster)                 VATIN/USt-ID:
Germany                                 DE305133015
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20201010/a2913271/attachment.sig>

Hi Philipp,

Edit (apologies for the repeated emails) Thanks for the quick response. I’ve run
the command and you can find the output by following this link
https://systrum.net/output.txt <https://systrum.net/output.txt>
I'm running icecast with SSL enabled. I needed a a version of Icecast with
SSL support which is why I am using the backports version.

Regards
Damian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20201010/233d3fd0/attachment.html>