Hi, needing some advice. I am running a backports version of icecast for Debian 10. I have found that with this version, the icecast server will usually run for about 7 days before it crashes. I found in the syslog that icecast gets terminated because it?s 'out of memory'. This never happened on the standard (stable) version of icecast that I ran on Debian 9 installed via apt. I?ve added the relevant section of the syslog below. Any help on steps to take would be greatly appreciated. Oct 10 10:36:40 systrum kernel: [925937.080802] 0 pages HighMem/MovableOnly Oct 10 10:36:40 systrum kernel: [925937.082050] 53826 pages reserved Oct 10 10:36:40 systrum kernel: [925937.083128] Tasks state (memory values in pages): Oct 10 10:36:40 systrum kernel: [925937.084663] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Oct 10 10:36:40 systrum kernel: [925937.087367] [ 227] 0 227 11670 1758 110592 0 0 systemd-journal Oct 10 10:36:40 systrum kernel: [925937.090230] [ 240] 0 240 5018 228 65536 0 -1000 systemd-udevd Oct 10 10:36:40 systrum kernel: [925937.093264] [ 312] 0 312 2110 70 53248 0 0 cron Oct 10 10:36:40 systrum kernel: [925937.096122] [ 313] 0 313 59770 289 98304 0 0 polkitd Oct 10 10:36:40 systrum kernel: [925937.098769] [ 314] 0 314 56433 391 77824 0 0 rsyslogd Oct 10 10:36:40 systrum kernel: [925937.111310] [ 315] 104 315 2220 198 57344 0 -900 dbus-daemon Oct 10 10:36:40 systrum kernel: [925937.114461] [ 316] 105 316 629 33 40960 0 0 nscd Oct 10 10:36:40 systrum kernel: [925937.117285] [ 318] 0 318 4849 256 73728 0 0 systemd-logind Oct 10 10:36:40 systrum kernel: [925937.120289] [ 319] 0 319 1608 56 49152 0 0 qemu-ga Oct 10 10:36:40 systrum kernel: [925937.123219] [ 400] 0 400 2372 326 53248 0 0 dhclient Oct 10 10:36:40 systrum kernel: [925937.125971] [ 489] 0 489 2372 324 53248 0 0 dhclient Oct 10 10:36:40 systrum kernel: [925937.128553] [ 592] 0 592 2372 322 57344 0 0 dhclient Oct 10 10:36:40 systrum kernel: [925937.131123] [ 663] 0 663 27551 1912 114688 0 0 unattended-upgr Oct 10 10:36:40 systrum kernel: [925937.133978] [ 665] 0 665 57651 2386 196608 0 0 php-fpm7.3 Oct 10 10:36:40 systrum kernel: [925937.136604] [ 669] 0 669 65100 2885 135168 0 0 tuned Oct 10 10:36:40 systrum kernel: [925937.139112] [ 686] 0 686 1316 16 49152 0 0 agetty Oct 10 10:36:40 systrum kernel: [925937.141673] [ 689] 0 689 641 18 40960 0 0 agetty Oct 10 10:36:40 systrum kernel: [925937.144260] [ 699] 106 699 19117 146 61440 0 0 ntpd Oct 10 10:36:40 systrum kernel: [925937.146907] [ 720] 33 720 57651 1770 176128 0 0 php-fpm7.3 Oct 10 10:36:40 systrum kernel: [925937.150091] [ 721] 33 721 57651 1770 176128 0 0 php-fpm7.3 Oct 10 10:36:40 systrum kernel: [925937.153211] [ 1119] 0 1119 10869 211 81920 0 0 master Oct 10 10:36:40 systrum kernel: [925937.156248] [ 1123] 110 1123 10892 231 81920 0 0 qmgr Oct 10 10:36:40 systrum kernel: [925937.159217] [ 22760] 0 22760 3942 214 69632 0 -1000 sshd Oct 10 10:36:40 systrum kernel: [925937.162157] [ 28701] 0 28701 304811 4989 262144 0 0 fail2ban-server Oct 10 10:36:40 systrum kernel: [925937.165398] [ 8095] 108 8095 1988579 1896082 15364096 0 0 icecast2 Oct 10 10:36:40 systrum kernel: [925937.168408] [ 8240] 1000 8240 152698 21447 344064 0 0 liquidsoap Oct 10 10:36:40 systrum kernel: [925937.171386] [ 8432] 1000 8432 151856 20770 331776 0 0 liquidsoap Oct 10 10:36:40 systrum kernel: [925937.174400] [ 17839] 0 17839 58512 2828 204800 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.177286] [ 30783] 33 30783 77224 3124 217088 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.180337] [ 10477] 33 10477 77171 3087 212992 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.183216] [ 13167] 33 13167 77184 3098 217088 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.186299] [ 18536] 33 18536 77172 3087 217088 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.189379] [ 21245] 33 21245 77164 3091 212992 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.192333] [ 24426] 33 24426 77195 3078 217088 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.195363] [ 24885] 33 24885 77169 3079 212992 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.198359] [ 25783] 33 25783 77180 3077 217088 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.201343] [ 28454] 33 28454 77169 3087 217088 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.204308] [ 29382] 33 29382 77178 3080 212992 0 0 apache2 Oct 10 10:36:40 systrum kernel: [925937.207252] [ 2631] 110 2631 10852 198 81920 0 0 pickup Oct 10 10:36:40 systrum kernel: [925937.210233] Out of memory: Kill process 8095 (icecast2) score 929 or sacrifice child Oct 10 10:36:40 systrum kernel: [925937.213229] Killed process 8095 (icecast2) total-vm:7954316kB, anon-rss:7584328kB, file-rss:0kB, shmem-rss:0kB -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xiph.org/pipermail/icecast/attachments/20201010/be2c5e5d/attachment.htm>
Philipp Schafft
2020-Oct-10 10:43 UTC
[Icecast] Icecast crashing / terminated - out of memory
Good morning, first of all, thank you for your nice report. :) On Sat, 2020-10-10 at 20:32 +1000, Damian wrote:> Hi, needing some advice. > I am running a backports version of icecast for Debian 10. I have > found that with this version, the icecast server will usually run for > about 7 days before it crashes. I found in the syslog that icecast > gets terminated because it?s 'out of memory'.There is currently a problem known in wolfssl which leads to Icecast leaking memory. See [0], [1].> This never happened on the standard (stable) version of icecast that I > ran on Debian 9 installed via apt. > I?ve added the relevant section of the syslog below. Any help on steps > to take would be greatly appreciated.That matches the picture. Do you have TLS enabled? Please provide a: cat /proc/$(pidof icecast2)/maps This allows us to verify if it's the known problem. You can run the command directly after you started Icecast. With best regards,> > Oct 10 10:36:40 systrum kernel: [925937.080802] 0 pages HighMem/MovableOnly > Oct 10 10:36:40 systrum kernel: [925937.082050] 53826 pages reserved > Oct 10 10:36:40 systrum kernel: [925937.083128] Tasks state (memory values in pages): > Oct 10 10:36:40 systrum kernel: [925937.084663] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name > [...] > Oct 10 10:36:40 systrum kernel: [925937.165398] [ 8095] 108 8095 1988579 1896082 15364096 0 0 icecast2 > [...] > Oct 10 10:36:40 systrum kernel: [925937.210233] Out of memory: Kill process 8095 (icecast2) score 929 or sacrifice child > Oct 10 10:36:40 systrum kernel: [925937.213229] Killed process 8095 (icecast2) total-vm:7954316kB, anon-rss:7584328kB, file-rss:0kB, shmem-rss:0kB >[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969018 [1] https://gitlab.xiph.org/xiph/icecast-server/-/issues/2391 -- Philipp Schafft (CEO/Gesch?ftsf?hrer) Telephon: +49.3535 490 17 92 L?wenfelsen UG (haftungsbeschr?nkt) Registration number: Bickinger Stra?e 21 HRB 12308 CB 04916 Herzberg (Elster) VATIN/USt-ID: Germany DE305133015 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: <http://lists.xiph.org/pipermail/icecast/attachments/20201010/a2913271/attachment.sig>
Hi Philipp, Edit (apologies for the repeated emails) Thanks for the quick response. I’ve run the command and you can find the output by following this link https://systrum.net/output.txt <https://systrum.net/output.txt> I'm running icecast with SSL enabled. I needed a a version of Icecast with SSL support which is why I am using the backports version. Regards Damian -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xiph.org/pipermail/icecast/attachments/20201010/233d3fd0/attachment.html>