asterisk users - Jul 2020 - 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable

Hi all

I'm running an Asterisk 13.22.0 instance on Centos 7 - I7-8700 12 core HT
with 16GB of RAM.

The server maintains a total active call count of approx 285 calls with 440
channels at any one time. The totals never go below 200 calls concurrently
active.

For the kernel, top reports load levels at around 5.0 to 6.0 constantly.

I'm having issues where at random intervals, the CLI is filled with
thousands of these lines

[Jul  1 14:32:24] WARNING[11530]: http.c:1942 httpd_helper_thread: HTTP
session count exceeded 100 sessions.

This appears to be some form of race condition, the Asterisk instance goes
completely unresponsive, all RTP stops and all calls get hung up the moment
these appear on the CLI. All registered SIP phones then deregister off the
server withing one to two minutes.

At the time this takes place, kernel load spikes to about 9.0 and then falls
off to under 1 while asterisk endlessly outputs the above warning in the
Asterisk CLI.

You can still log into the CLI but no command is executed, you also cannot
get the Asterisk to stop with HUP / kill -1 and I have to kill -9 the
Asterisk PID at the end of the day.

I'm also, just prior the above taking place, getting tens of thousands of
these warnings in the CLI:

[Jul  1 14:30:39] WARNING[16819][C-000004bf]: taskprocessor.c:895
taskprocessor_push: The 'stasis-core-control' task processor queue
reached
500 scheduled tasks again.

I'm using AMI at several hundred lines of traffic per second to control the
Asterisk instance, spawn new calls, get queue and agent statuses, etc. -
standard AMI stuff.

Where can I start to look for how to fix this "session count exceeded 100
sessions" error?

Sometimes, it doesn't come up for weeks (loads remain much the same) and
then on one day it will happen four to six times in quick succession, load
broadly the same.

Any ideas?

Thanks!

Stefan

On Wed, Jul 1, 2020 at 9:49 AM Stefan Viljoen <viljoens at
verishare.co.za>
wrote:
> Hi all
>
> I'm running an Asterisk 13.22.0 instance on Centos 7 - I7-8700 12 core
HT
> with 16GB of RAM.
>
> The server maintains a total active call count of approx 285 calls with 440
> channels at any one time. The totals never go below 200 calls concurrently
> active.
>
> For the kernel, top reports load levels at around 5.0 to 6.0 constantly.
>
> I'm having issues where at random intervals, the CLI is filled with
> thousands of these lines
>
> [Jul  1 14:32:24] WARNING[11530]: http.c:1942 httpd_helper_thread: HTTP
> session count exceeded 100 sessions.
>
> This appears to be some form of race condition, the Asterisk instance goes
> completely unresponsive, all RTP stops and all calls get hung up the moment
> these appear on the CLI. All registered SIP phones then deregister off the
> server withing one to two minutes.
>
This isn't really a race condition. The message occurs when the limit for
simultaneous HTTP sessions is exceeded. You haven't explained what the HTTP
server in Asterisk is being used for, so I'd start with investigating that.
Is ARI being used? WebRTC? If you do a packet capture on it do you see a
flood of incoming HTTP connection attempts?

-- 
Joshua C. Colp
Asterisk Technical Lead
Sangoma Technologies
Check us out at www.sangoma.com and www.asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20200701/c0f1dd1f/attachment.html>

Hi Joshua

HTTP is used on in our setup on

127.0.0.1/mxml?<command>

to send commands to the server, such as

http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret

to log in and then

http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1

etc. to control transfers, for example.

ARI is not being used.

WebRTC is not being used.

I have in the meantime modified httpd.conf and halved the session_inactivity
from 30 000 to 20 000 and session_keep_alive from 15 000 to 10 000, and
increased the sessionlimit to 1000 (instead of 100) to see if shorter sessions
expiring more frequently and just plain more available slots for sessions will
help?

So far so good, done about 55 000 new calls that way the past 32 minutes.

Or am I barking up the wrong tree?

Thanks!

--
Stefan