asterisk users - Jul 2020 - 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable

Hi Joshua

HTTP is used on in our setup on

127.0.0.1/mxml?<command>

to send commands to the server, such as

http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret

to log in and then

http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1

etc. to control transfers, for example.

ARI is not being used.

WebRTC is not being used.

I have in the meantime modified httpd.conf and halved the session_inactivity
from 30 000 to 20 000 and session_keep_alive from 15 000 to 10 000, and
increased the sessionlimit to 1000 (instead of 100) to see if shorter sessions
expiring more frequently and just plain more available slots for sessions will
help?

So far so good, done about 55 000 new calls that way the past 32 minutes.

Or am I barking up the wrong tree?

Thanks!

--
Stefan

On Wed, Jul 1, 2020 at 10:32 AM Stefan Viljoen <viljoens at
verishare.co.za>
wrote:
> Hi Joshua
>
> HTTP is used on in our setup on
>
> 127.0.0.1/mxml?<command>
>
> to send commands to the server, such as
>
> http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret
>
> to log in and then
>
>
>
http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1
>
> etc. to control transfers, for example.
>
> ARI is not being used.
>
> WebRTC is not being used.
>
> I have in the meantime modified httpd.conf and halved the
> session_inactivity from 30 000 to 20 000 and session_keep_alive from 15 000
> to 10 000, and increased the sessionlimit to 1000 (instead of 100) to see
> if shorter sessions expiring more frequently and just plain more available
> slots for sessions will help?
>
> So far so good, done about 55 000 new calls that way the past 32 minutes.
>
> Or am I barking up the wrong tree?
>
It's possible it will help - but that still doesn't explain what exactly
happened. Does your code have a back off strategy if HTTP requests fail? If
not and they have an immediate retry with loop, then that may have
effectively DDoSed things. It would also be good to explain why 100
connections were in use - did old connections not get closed? Was your load
sufficient to just meet that amount?

-- 
Joshua C. Colp
Asterisk Technical Lead
Sangoma Technologies
Check us out at www.sangoma.com and www.asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20200701/02e1bee1/attachment.html>

Hi Joshua

No back-off, but I am caching the last 5000 result and and first hitting the
cache to see if a recent command already provided the information I'm
seeking for a particular request.




From: Joshua C. Colp <jcolp at sangoma.com> 
Sent: Wednesday, 01 July 2020 15:47
To: viljoens at verishare.co.za; Asterisk Users Mailing List - Non-Commercial
Discussion <asterisk-users at lists.digium.com>
Subject: Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions
- instance unusable

On Wed, Jul 1, 2020 at 10:32 AM Stefan Viljoen <mailto:viljoens at
verishare.co.za> wrote:
Hi Joshua

HTTP is used on in our setup on

http://127.0.0.1/mxml?<command>

to send commands to the server, such as

http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret

to log in and then

http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1

etc. to control transfers, for example.

ARI is not being used.

WebRTC is not being used.

I have in the meantime modified httpd.conf and halved the session_inactivity
from 30 000 to 20 000 and session_keep_alive from 15 000 to 10 000, and
increased the sessionlimit to 1000 (instead of 100) to see if shorter sessions
expiring more frequently and just plain more available slots for sessions will
help?

So far so good, done about 55 000 new calls that way the past 32 minutes.

Or am I barking up the wrong tree?

It's possible it will help - but that still doesn't explain what exactly
happened. Does your code have a back off strategy if HTTP requests fail? If not
and they have an immediate retry with loop, then that may have effectively
DDoSed things. It would also be good to explain why 100 connections were in use
- did old connections not get closed? Was your load sufficient to just meet that
amount?

-- 
Joshua C. Colp
Asterisk Technical Lead
Sangoma Technologies
Check us out at http://www.sangoma.com and http://www.asterisk.org

Hi Joshua

No back-off, but I am caching the last 5000 results and and first hitting the
cache to see if a recent command already provided the information I'm
seeking for a particular request.

I'll see if I can do some simulation and see if I'm effectively DDOSing
the local HTTP interface.

I'll have to see if I maybe have a resource leak in my code that makes the
HTTP request, which is set  to "Connection: close" already (and not
"Connection: keep-alive" at HTTP request header level) hang around and
keep the HTTP connection open to Asterisk.

Does the HTTP implementation in Asterisk heed this "Connection:" HTTP
header?

--
Stefan