Hi, ever since I have switched my server from Centos 7 to Fedora 32, asterisk is showing memory issues and no calls are possible. I'm using the asterisk that comes with Fedora; before that, I used a self-compiled version on Centos. The hardware is still the same. Asterisk shows the following message when trying to make a call: WARNING[92530]: pbx.c:4644 increase_call_count: Available system memory (~168MB) is below the configured low watermark (1024MB) I was thinking that asterisk is leaking memory because the only way to get asterisk to work again was by restarting the server. Since this is very annoying and the Fedora bug report remains ignored, I finally started to investigate. The relevant source is attached in asnip.c. To see what's going on, I wrote a test program, attached as sysinfo.c. You can simply compile it with 'cc -O2 sysinfo.c -o sysinfo'. The output is follows: ./sysinfo unit size: 1 byte(s) # freeram: 153 bufferram: 5 ---- Sum: 158 # trying to allocate 158 sleeping 5 seconds allocate twice as much sleeping 5 seconds memset allocated memory to 0 sleeping 5 seconds memory freed So what asterisk says is about right. When I look at the info from 'cat /proc/meminfo', I see this: cat /proc/meminfo MemTotal: 16361780 kB MemFree: 151844 kB MemAvailable: 14883060 kB Buffers: 5468 kB Cached: 14773136 kB SwapCached: 224 kB Active: 2019568 kB Inactive: 13592124 kB [...] which would mean that I have 14GB buffered/cached, and 'free -h' confirms this. Apparently, the cache remains persistently occupied. Since I'm currently performing backups, it's not surprising that the cache is large. Apparently it means that asterisk fails every time I'm doing something that fills the cache :( After understanding that the cache remains full, I figured there might be way to flush the buffers and the cache. [1] shows how to do this, and after 'sync; echo 3 > /proc/sys/vm/drop_caches', 'free -h' showed buff/cache as 1.3Gi and asterisk was working again. However, the backups are still going on, and it doesn't take long before the cache is back at 14GB again and asterisk is blocked. I think asterisk needs to consider the cache as free memory as well. Isn't the cache supposed to automatically shrink when more memory is required? As a workaround, I can set minmemfree in asterisk.conf to a low value. Nonetheless, I guess that should be fixed. I'll update the Fedora bug report with this. [1]: https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/ -------------- next part -------------- A non-text attachment was scrubbed... Name: asnip.c Type: text/x-csrc Size: 660 bytes Desc: not available URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20200925/a9304b00/attachment.c> -------------- next part -------------- A non-text attachment was scrubbed... Name: sysinfo.c Type: text/x-csrc Size: 1522 bytes Desc: not available URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20200925/a9304b00/attachment-0001.c>
https://issues.asterisk.org/jira/browse/ASTERISK-28695 On Fri, Sep 25, 2020 at 2:32 PM hw <hw at gc-24.de> wrote:> > > Hi, > > > > ever since I have switched my server from Centos 7 to Fedora 32, asterisk > > is showing memory issues and no calls are possible. I'm using the asterisk > > that comes with Fedora; before that, I used a self-compiled version on > > Centos. The hardware is still the same. Asterisk shows the following > > message when trying to make a call: > > > > > > WARNING[92530]: pbx.c:4644 increase_call_count: Available system memory > > (~168MB) is below the configured low watermark (1024MB) > > > > > > I was thinking that asterisk is leaking memory because the only way to get > > asterisk to work again was by restarting the server. Since this is very > > annoying and the Fedora bug report remains ignored, I finally started to > > investigate. The relevant source is attached in asnip.c. > > > > To see what's going on, I wrote a test program, attached as sysinfo.c. You > > can simply compile it with 'cc -O2 sysinfo.c -o sysinfo'. > > > > The output is follows: > > > > > > ./sysinfo > > unit size: 1 byte(s) > > # > > freeram: 153 > > bufferram: 5 > > ---- > > Sum: 158 > > # > > trying to allocate 158 > > sleeping 5 seconds > > allocate twice as much > > sleeping 5 seconds > > memset allocated memory to 0 > > sleeping 5 seconds > > memory freed > > > > > > So what asterisk says is about right. When I look at the info from 'cat > > /proc/meminfo', I see this: > > > > > > cat /proc/meminfo > > MemTotal: 16361780 kB > > MemFree: 151844 kB > > MemAvailable: 14883060 kB > > Buffers: 5468 kB > > Cached: 14773136 kB > > SwapCached: 224 kB > > Active: 2019568 kB > > Inactive: 13592124 kB > > [...] > > > > > > which would mean that I have 14GB buffered/cached, and 'free -h' confirms > > this. > > > > Apparently, the cache remains persistently occupied. Since I'm currently > > performing backups, it's not surprising that the cache is large. > > Apparently it means that asterisk fails every time I'm doing something > > that fills the cache :( > > > > After understanding that the cache remains full, I figured there might be > > way to flush the buffers and the cache. [1] shows how to do this, and > > after 'sync; echo 3 > /proc/sys/vm/drop_caches', 'free -h' showed > > buff/cache as 1.3Gi and asterisk was working again. > > > > However, the backups are still going on, and it doesn't take long before > > the cache is back at 14GB again and asterisk is blocked. > > > > I think asterisk needs to consider the cache as free memory as well. Isn't > > the cache supposed to automatically shrink when more memory is required? > > > > As a workaround, I can set minmemfree in asterisk.conf to a low value. > > Nonetheless, I guess that should be fixed. I'll update the Fedora bug > > report with this. > > > > > > [1]: > > > https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/ > > > > -- > > _____________________________________________________________________ > > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > > > Check out the new Asterisk community forum at: > https://community.asterisk.org/ > > > > New to Asterisk? Start here: > > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > > > asterisk-users mailing list > > To UNSUBSCRIBE or update options visit: > > http://lists.digium.com/mailman/listinfo/asterisk-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20200925/d4bf1b2b/attachment.html>
On Fri, 2020-09-25 at 21:32 -0400, Sean Bright wrote:> https://issues.asterisk.org/jira/browse/ASTERISK-28695 >Thanks! The fix doesn't fix it because the cache must be considered; the bufferram isn't so relevant. A few kB more doesn't make much difference. Could/should I re-open this bug report?
Reasonably Related Threads
- memory issues
- [PATCH v2 1/1] virtio_balloon: include buffers and cached memory statistics
- [PATCH v2 1/1] virtio_balloon: include buffers and cached memory statistics
- [PATCH v2 1/1] virtio_balloon: include buffers and cached memory statistics
- [PATCH v2 1/1] virtio_balloon: include buffers and cached memory statistics