<div>Hello</div><div>I'm trying to measure the performance of the icecast relay server on 64kbps streams.</div><div> </div><div>The server is running in AWS (i've tried various instance types) and the test clients are running on other machines in AWS. The test client is a very simple libuv application that sends a GET request and basically ignores everything it receives in the response. I'm using the icecast-kh fork.</div><div> </div><div>I'm able to go up to around 9K simultaneous connections to the server (from two machines). The CPU usage is low, about 15% or so (on one core). However connections are starting to be dropped. Checking netstat i see many frames being lost. Increasing the transmit queue length helped, but still i can't go beyond around 9K connections. I have increased the file descriptor limits and configured IRQ balancing (even though the problem doesn't seem to be CPU bound)</div><div> </div><div>It doesn't matter if i run one or more instances of the relay server, the limit seems to be OS global so when one instance is running with 5K connections and the other instance is getting close to 4K connections they both start dropping connections.</div><div> </div><div>I assume that there is some other setting of the stack that i didn't configure so i was wondering if anybody was able to run a few dozens of thousands of connections on one server. </div><div> </div><div>thanks!</div><div>--zahar</div>
Hi Zahar, what value has into
<workers>8</workers>
This value is recommended to set at 1 by virtual core.
I 'm using for some years, KH branch with 35 concurrent listeners into 8
core vm.
Regards,
Alejandro
2016-06-10 0:50 GMT-03:00 Zahar Popov <zahar.popov1978 at yandex.com>:
> Hello
> I'm trying to measure the performance of the icecast relay server on
> 64kbps streams.
>
> The server is running in AWS (i've tried various instance types) and
the
> test clients are running on other machines in AWS. The test client is a
> very simple libuv application that sends a GET request and basically
> ignores everything it receives in the response. I'm using the
icecast-kh
> fork.
>
> I'm able to go up to around 9K simultaneous connections to the server
> (from two machines). The CPU usage is low, about 15% or so (on one core).
> However connections are starting to be dropped. Checking netstat i see many
> frames being lost. Increasing the transmit queue length helped, but still i
> can't go beyond around 9K connections. I have increased the file
descriptor
> limits and configured IRQ balancing (even though the problem doesn't
seem
> to be CPU bound)
>
> It doesn't matter if i run one or more instances of the relay server,
the
> limit seems to be OS global so when one instance is running with 5K
> connections and the other instance is getting close to 4K connections they
> both start dropping connections.
>
> I assume that there is some other setting of the stack that i didn't
> configure so i was wondering if anybody was able to run a few dozens of
> thousands of connections on one server.
>
> thanks!
> --zahar
>
> _______________________________________________
> Icecast-dev mailing list
> Icecast-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast-dev/attachments/20160610/4922f570/attachment.html>
Sorry, 35K concurrent with 8 workers at 60% cpu 2016-06-10 1:28 GMT-03:00 Alejandro <cdgraff at gmail.com>:> Hi Zahar, what value has into > > <workers>8</workers> > > This value is recommended to set at 1 by virtual core. > > I 'm using for some years, KH branch with 35 concurrent listeners into 8 > core vm. > > Regards, > Alejandro > > 2016-06-10 0:50 GMT-03:00 Zahar Popov <zahar.popov1978 at yandex.com>: > >> Hello >> I'm trying to measure the performance of the icecast relay server on >> 64kbps streams. >> >> The server is running in AWS (i've tried various instance types) and the >> test clients are running on other machines in AWS. The test client is a >> very simple libuv application that sends a GET request and basically >> ignores everything it receives in the response. I'm using the icecast-kh >> fork. >> >> I'm able to go up to around 9K simultaneous connections to the server >> (from two machines). The CPU usage is low, about 15% or so (on one core). >> However connections are starting to be dropped. Checking netstat i see many >> frames being lost. Increasing the transmit queue length helped, but still i >> can't go beyond around 9K connections. I have increased the file descriptor >> limits and configured IRQ balancing (even though the problem doesn't seem >> to be CPU bound) >> >> It doesn't matter if i run one or more instances of the relay server, the >> limit seems to be OS global so when one instance is running with 5K >> connections and the other instance is getting close to 4K connections they >> both start dropping connections. >> >> I assume that there is some other setting of the stack that i didn't >> configure so i was wondering if anybody was able to run a few dozens of >> thousands of connections on one server. >> >> thanks! >> --zahar >> >> _______________________________________________ >> Icecast-dev mailing list >> Icecast-dev at xiph.org >> http://lists.xiph.org/mailman/listinfo/icecast-dev >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xiph.org/pipermail/icecast-dev/attachments/20160610/b16f9741/attachment.html>
Philipp Schafft
2016-Jun-10 10:09 UTC
[Icecast-dev] icecast relay server performance testing
Good noon, On Fri, 2016-06-10 at 12:50 +0900, Zahar Popov wrote:> Hello > I'm trying to measure the performance of the icecast relay server on > 64kbps streams. > > The server is running in AWS [...]. I'm using the icecast-kh fork. > [...] > I'm able to go up to around 9K simultaneous connections to the server > (from two machines). [...]First: -kh is a independent fork. So I can hardly speak for it. This answer is completely based on the (often wrong) assumption that in this case behaves exactly like the official Icecast2 in any stable and supported version.> It doesn't matter if i run one or more instances of the relay server, > the limit seems to be OS global so when one instance is running with > 5K connections and the other instance is getting close to 4K > connections they both start dropping connections. > > I assume that there is some other setting of the stack that i didn't > configure so i was wondering if anybody was able to run a few dozens > of thousands of connections on one server.This should give you the hint: It's not a problem of Icecast2 as it doesn't depend on the processes. Basically: AWS is well known for their broken TCP stack. There is no reason to run Icecast2 on AWS. Just use a normal server and be fine. What you see above is just AWS's maximum connection limit per instance or something. (But there are more problems known with AWS's TCP stack that affect real world deployments of Icecast2.) Have a nice day! -- Philipp. (Rah of PH2) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: This is a digitally signed message part URL: <http://lists.xiph.org/pipermail/icecast-dev/attachments/20160610/b33ada96/attachment.sig>
Hi Philipp Thank you for chiming in. The only reason i use the -kh fork is because it seemed to be more recent. I had this performance issue with the regular version and tried the fork. I realize that the problem can be with the TCP stack parameters. I had no problem getting about 800Mbps between these machines when using iperf. Certainly, the workload completely different, but at least i know that the TCP stack is somewhat operational. Do you happen to know specifically what’s broken? I don’t have access to a physical data center this is why i would like to use either EC2 or Azure. thanks! —zahar> On Jun 10, 2016, at 7:09 PM, Philipp Schafft <lion at lion.leolix.org> wrote: > > Good noon, > > On Fri, 2016-06-10 at 12:50 +0900, Zahar Popov wrote: >> Hello >> I'm trying to measure the performance of the icecast relay server on >> 64kbps streams. >> >> The server is running in AWS [...]. I'm using the icecast-kh fork. >> [...] >> I'm able to go up to around 9K simultaneous connections to the server >> (from two machines). [...] > > First: -kh is a independent fork. So I can hardly speak for it. This > answer is completely based on the (often wrong) assumption that in this > case behaves exactly like the official Icecast2 in any stable and > supported version. > > >> It doesn't matter if i run one or more instances of the relay server, >> the limit seems to be OS global so when one instance is running with >> 5K connections and the other instance is getting close to 4K >> connections they both start dropping connections. >> >> I assume that there is some other setting of the stack that i didn't >> configure so i was wondering if anybody was able to run a few dozens >> of thousands of connections on one server. > > This should give you the hint: It's not a problem of Icecast2 as it > doesn't depend on the processes. > > Basically: AWS is well known for their broken TCP stack. There is no > reason to run Icecast2 on AWS. Just use a normal server and be fine. > What you see above is just AWS's maximum connection limit per instance > or something. (But there are more problems known with AWS's TCP stack > that affect real world deployments of Icecast2.) > > > Have a nice day! > > -- > Philipp. > (Rah of PH2)