Icecast - Mar 2005 - high CPU load for large # sources?

Hi all,

I have an icecast setup with 20+ sources. During peak times some 20 sources
will be connected with a total of some 250 listeners more-or-less equally
divided over the 20 sources. All streams are running at a measly 16 kbps.
There is enough bandwidth to/from the server. During these peak times I see
very high CPU usage for icecast 98-99%. The system I'm running is an Intel
Celeron 2.4GHz running Fedora Linux 1. It does appear that the server is
happily serverving all clients, but what could cause the high CPU load?

Is every source (/mount) serviced by a different thread? How many threads
would I have then? My threadpool is set at 20. Could it be that the timeout
on poll/select of 250 in each thread causes Icecast to effectively busywait
since with 20 threads the actual time between the poll/select system calls
is reduced to ~10ms, the timeslice of Linux!

Here's the relevant part of my configuration file.

 <limits>
        <clients>2000</clients>
        <sources>100</sources>
        <threadpool>20</threadpool>
        <queue-size>512000</queue-size>
        <client-timeout>30</client-timeout>
        <header-timeout>30</header-timeout>
        <source-timeout>30</source-timeout>
        <burst-on-connect>1</burst-on-connect>
        <burst-size>20480</burst-size>
    </limits>


Has anyone else experienced high CPU load with Icecast?

Cheers,
KJ

On Mon, 2005-03-07 at 22:01, Klaas Jan Wierenga wrote:
> Hi all,
> 
> I have an icecast setup with 20+ sources. During peak times some 20 sources
> will be connected with a total of some 250 listeners more-or-less equally
> divided over the 20 sources. All streams are running at a measly 16 kbps.
> There is enough bandwidth to/from the server. During these peak times I see
> very high CPU usage for icecast 98-99%. The system I'm running is an
Intel
> Celeron 2.4GHz running Fedora Linux 1. It does appear that the server is
> happily serverving all clients, but what could cause the high CPU load?
I've only seen a busy CPU report on OS X which was relating to the file
serving.
> Is every source (/mount) serviced by a different thread? How many threads
> would I have then? My threadpool is set at 20. Could it be that the timeout
> on poll/select of 250 in each thread causes Icecast to effectively busywait
> since with 20 threads the actual time between the poll/select system calls
> is reduced to ~10ms, the timeslice of Linux!
Each source is handled by a separate thread, but there's no relationship
between those and threadpool, it's unlikely that a thread pool of more
than 2 or 3 is ever useful.

If poll/select are returning continuously before 250 ms then it's either
signal related (unlikely) or some fd has triggered, the poll timeout can
also drop to 0 on client bursts, but those are short lived.

I would determine which thread(s) are busy from ps, then use 

strace -o output.txt -p <thread of busy icecast>

ctrl-C it after a few seconds, then email the output.txt file to me.

karl.

Ok Karl,

Thanks for your help, I'll send the strace output after the next 'peak
time'
:-).
You reply did ring a bell with me. I have fileserving on and it is also not
unlikely that someone is downloading a recorded broadcast.... silly me. I'll
investigate further when the situation arises again.

Cheers,
KJ

-----Oorspronkelijk bericht-----
Van: Karl Heyes [mailto:karl@xiph.org]
Verzonden: maandag 7 maart 2005 23:31
Aan: Klaas Jan Wierenga
CC: icecast
Onderwerp: Re: [Icecast] high CPU load for large # sources?


On Mon, 2005-03-07 at 22:01, Klaas Jan Wierenga wrote:
> Hi all,
>
> I have an icecast setup with 20+ sources. During peak times some 20
sources
> will be connected with a total of some 250 listeners more-or-less equally
> divided over the 20 sources. All streams are running at a measly 16 kbps.
> There is enough bandwidth to/from the server. During these peak times I
see
> very high CPU usage for icecast 98-99%. The system I'm running is an
Intel
> Celeron 2.4GHz running Fedora Linux 1. It does appear that the server is
> happily serverving all clients, but what could cause the high CPU load?
I've only seen a busy CPU report on OS X which was relating to the file
serving.
> Is every source (/mount) serviced by a different thread? How many threads
> would I have then? My threadpool is set at 20. Could it be that the
timeout
> on poll/select of 250 in each thread causes Icecast to effectively
busywait
> since with 20 threads the actual time between the poll/select system calls
> is reduced to ~10ms, the timeslice of Linux!
Each source is handled by a separate thread, but there's no relationship
between those and threadpool, it's unlikely that a thread pool of more
than 2 or 3 is ever useful.

If poll/select are returning continuously before 250 ms then it's either
signal related (unlikely) or some fd has triggered, the poll timeout can
also drop to 0 on client bursts, but those are short lived.

I would determine which thread(s) are busy from ps, then use

strace -o output.txt -p <thread of busy icecast>

ctrl-C it after a few seconds, then email the output.txt file to me.

karl.