Icecast - Nov 2020 - help with "too many clients awaiting authentication"

Hello guys,

Today we faced an unique problem.

We have about 120 stations and we are reaching 40k simultaneous listeners
monday/friday (~ 11 AM GMT -3).

We also use url auth to log, audit and build audience reports.

But today, at the end of the day we were about 10k simultaneous and listeners
start get errors (403, busy, please try again later), and after digging, delete
logs and restart icecast we found this:

[2020-11-15  22:09:38] INFO auth/queue_auth_client auth on (default mount) has
101 pending
[2020-11-15  22:09:38] WARN auth/auth_add_listener too many clients awaiting
authentication

Before those errors our auth api was logging ~60 rps in ~150ms (during the error
the api was not getting requests).

Has anyone else has already faced this problem? Any tips?

Best,
Thiago
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20201115/0311f293/attachment.html>

Good morning,

hope you're fine. :)


On Sun, 2020-11-15 at 22:12 -0300, subscription at nextdial.com.br
wrote:
> Hello guys,
> 
> Today we faced an unique problem.
> 
> We have about 120 stations and we are reaching 40k simultaneous
> listeners monday/friday (~ 11 AM GMT -3).
> 
> We also use url auth to log, audit and build audience reports.
> 
> But today, at the end of the day we were about 10k simultaneous and
> listeners start get errors (403, busy, please try again later), and
> after digging, delete logs and restart icecast we found this:
> 
> [2020-11-15  22:09:38] INFO auth/queue_auth_client auth on (default
> mount) has 101 pending
> [2020-11-15  22:09:38] WARN auth/auth_add_listener too many clients
> awaiting authentication
> 
Generally this happens if the queue of to-process clients reach it's
capacity. Generally this is a sign that the backend can not handle the
given load of clients.

In case of URL auth this normally means that the auth server does not
answer as fast as needed.


> Before those errors our auth api was logging ~60 rps in ~150ms (during
> the error the api was not getting requests).
Maybe the request at the head of the queue (the one currently
processing) was hanging for some reason?

> Has anyone else has already faced this problem? Any tips?
Did this happen again or just once?

Did the problem go away after a few seconds? The timeout for URL auth is
set to 15 seconds. Which might be a bit long in your case.

If it stabilized after a moment you could consider changing the timeout.
In our version you can find it in src/auth_url.c line 744:
        curl_easy_setopt (url_info->handle, CURLOPT_TIMEOUT, 15L);

Maybe change it to something like 3L.

Also: Did you do any maintenance on the infrastructure at the same time?
E.g. the server could not be reached for a second, or DNS was updated or
something?


If it happens again AND does not fix itself after a moment we should
have a closer look at it.


With best regards,

-- 
Philipp Schafft (CEO/Geschäftsführer) 
Telephon: +49.3535 490 17 92

Löwenfelsen UG (haftungsbeschränkt)     Registration number:
Bickinger Straße 21                     HRB 12308 CB
04916 Herzberg (Elster)                 VATIN/USt-ID:
Germany                                 DE305133015
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20201116/ee177f03/attachment.sig>