Icecast dev - Jul 2005 - icecast performance on many concurrentlow-bitrate streams

Hi Karl,

Thanks for your info. I have a standard Icecast-2.2 release with a few local
patches. I'm a little apprehesive to apply my patches to the kh14 branch, so
I'd rather patch my branch with the changes related to batched reads from
the kh branch.
I've looked at your code to see if I could spot the changes related to
batching reads. So far I have not been able to find where you've made this
patch. Could you point me in the right direction?

Cheers,
KJ

-----Oorspronkelijk bericht-----
Van: Karl Heyes [mailto:karl@xiph.org]
Verzonden: woensdag 27 juli 2005 16:47
Aan: Klaas Jan Wierenga
CC: icecast-dev
Onderwerp: Re: [Icecast-dev] icecast performance on many
concurrentlow-bitrate streams


On Wed, 2005-07-27 at 09:47, Klaas Jan Wierenga wrote:
> Hi all,
>
> I'm running an Icecast-2.2 server with at peak times some 50 sources
and
500 concurrent listeners all using low-bitrate 16kpbs streams. I'm
experiencing some connection losses at these peak times ("Client connection
died" message in error.log).
>
> The machine running Icecast has a 100Mbit connection to the internet. It
is a Celeron 2.4Ghz machine with 1Gbyte of main memory. The CPU load at
these peak times is normal at 0.4 (40%), the load is relatively high,
averaging out at 0.4 with occasional peaks to 5.0.
>
> I've analysed the ethernet packets on some of the listeners connections
and found that Icecast sends many small packets (200-300) bytes. This led me
to look at the interrupt rate during peak times. At these times the
interrupt rate reaches 10000 interrupts per second.
>
> Investigating a bit further I discovered that Icecast is turning off the
Nagle algorithm by setting TCP_NODELAY option on the client sockets. This
results in many small packets because a packet is sent as soon as possible
without combining packets into larger packets. Would it be safe to turn on
the Nagle algorithm (by removing the sock_set_nodelay() calls in appropriate
places) to try to reduce the interrupt rate for many concurrent low-bitrate
streams?

This has been reported to me already, it occurs with low bitrate non-ogg
streams.  You can remove the sock_set_nodelay as I don't think it really
does anything for us at all, but it may not help you either as it will
be up to the kernel to send the packets so it may just send those small
reads anyway (depends on various factors).

In kh14, I have done some batching up of reads on the input for
pass-through streams (like mp3/aac etc). This makes the writes to
listeners work on larger data (nearer to 1500 bytes) so protocol
overhead is made lower.

The feedback I've had so far is positive so it could be merged into
trunk without problem.  feel free to try it out

karl.

On Thu, 2005-07-28 at 15:30, Klaas Jan Wierenga wrote:
> Hi Karl,
> 
> Thanks for your info. I have a standard Icecast-2.2 release with a few
local patches. I'm a little apprehesive to apply my patches to the kh14
branch, so I'd rather patch my branch with the changes related to batched
reads from the kh branch.
> I've looked at your code to see if I could spot the changes related to
batching reads. So far I have not been able to find where you've made this
patch. Could you point me in the right direction?
The changes for batching part are really just isolated to the
format_mp3.[ch] files, within the 2 readers calls (filter_meta,
no_meta).

commit diff is listed on 
http://lists.xiph.org/pipermail/commits/2005-June/007469.html

There have been various changes between 2.2 and the trunk/kh code so I'm
sure that just dropping those 2 files into the 2.2.0 tree will not work
right off. For one thing the response headers are treated differently

I don't know the scope of your patches, so I can't even give you any
hints on those.


karl.

Karl,

I've managed to patch up my branch of Icecast to do the batching. Checked
everything with valgrind and tested it extensively. It looks good. Tcpdump
now shows nice size frames (mostly 1400 bytes). Any reason why you're not
settings the MTU to something closer to 1500?

Many thanks for your help,
KJ

-----Oorspronkelijk bericht-----
Van: Karl Heyes [mailto:karl@xiph.org]
Verzonden: donderdag 28 juli 2005 17:03
Aan: Klaas Jan Wierenga
CC: icecast-dev
Onderwerp: Re: [Icecast-dev] icecast performance on
manyconcurrentlow-bitrate streams


On Thu, 2005-07-28 at 15:30, Klaas Jan Wierenga wrote:
> Hi Karl,
>
> Thanks for your info. I have a standard Icecast-2.2 release with a few
local patches. I'm a little apprehesive to apply my patches to the kh14
branch, so I'd rather patch my branch with the changes related to batched
reads from the kh branch.
> I've looked at your code to see if I could spot the changes related to
batching reads. So far I have not been able to find where you've made this
patch. Could you point me in the right direction?

The changes for batching part are really just isolated to the
format_mp3.[ch] files, within the 2 readers calls (filter_meta,
no_meta).

commit diff is listed on
http://lists.xiph.org/pipermail/commits/2005-June/007469.html

There have been various changes between 2.2 and the trunk/kh code so I'm
sure that just dropping those 2 files into the 2.2.0 tree will not work
right off. For one thing the response headers are treated differently

I don't know the scope of your patches, so I can't even give you any
hints on those.


karl.