Hi Thomas, I don't know if you like top or bottom quoting. That seems to be a big-little endian thing ;-) So, I will top quote and inline quote. Please see my comments inline, below. Kind regards, Stephan On 02/19/2015 04:18 PM, "Thomas B. R?cker" wrote:> Hi, > > On 02/19/2015 03:07 PM, Stephan Leemburg wrote: >> Hello Icecast-dev, >> >> I am new to this list. > > Welcome!Thank you.> >> I am working for the NPO, the Dutch Public Broadcasting agency. >> We do a lot of icecast streaming. We run at least 20 icecast server >> instances on our media streaming cluster. > > That's very nice to hear. > >> We ran into an issue that clients which where connecting to our streams >> seemed to be 'hanging' on the connection setup frequently. The client >> 'thinks' it is connected, but no data. >> >> People suggested that it probably had to do with the poll() call. So, I >> looked into that. >> >> I found that the issue was actually caused by the very low listen >> backlog (5). >> >> On our clusters, we typically set this to 8192. Yes it is high, but we >> do a _lot_ of streaming and host very high volume websites. > > I'm not very familiar with socket programming, so will let Philipp > comment on this. Interesting enough this issue hasn't come up so far as > far as I can tell and there are some pretty high load deployments out there.We often have 'bursts' of new connections. Due to something said on a website, radio, tv, some top-2000 end of year event, etc. And when we get a lot of simultanious connection requests, this becomes an issue. I wrote a small server to simulate it. And the linux backlog acts a little bit different then expected.> > If you can share that info, what sort of concurrent listener load are we > talking about?Sure I can share, we are a national broadcasting agency funded by tax money. So no secrets here ;-) I just asked the media streaming guys.. 70k icecast connections on a regular day and 150k around special broadcasts (like top 2000 around new year).> > >> Currently we are using icecast 2.3. We are migrating to 2.4. >> So, I have written patches for 2.3 and 2.4, but also for the current 2.5 >> git tree. >> >> Unfortunately, I am a newby when it comes to git (sorry). But I do have >> unified diff patch files for the 2.3, 2.4 and 2.5 source trees. >> >> The patched 2.4 icecast was tested by our media streaming team and they >> confirmed that their issue was solved by it. >> >> Can I submit them (and how)? > > Just send them as attachments to this list, or open a ticket over at > https://trac.xiph.org. > If trac is naughty and thinks you're a spammer, please let me know. > > If there are differences between them, then please the 2.5 and 2.4 > patches. Otherwise 2.5 will do just fine. >I have attached the 2.4 and 2.5 patches.> > Thanks a lot for taking the time to reach out to us!Thank you for your Open Source contributions and efforts! And as it is Open Source, we can fix problems ourself and share.> > > Cheers > > Thomas > > _______________________________________________ > Icecast-dev mailing list > Icecast-dev at xiph.org > http://lists.xiph.org/mailman/listinfo/icecast-dev >-- Met vriendelijke groet, Stephan Leemburg, internetbeheerder NPO ICT Internet Services Bart de Graaffweg 2, 1217 ZL Hilversum servicedesk at omroep.nl, 035-6773555 -------------- next part -------------- A non-text attachment was scrubbed... Name: icecast-2.4.1-somaxconn.patch Type: text/x-patch Size: 4507 bytes Desc: not available Url : http://lists.xiph.org/pipermail/icecast-dev/attachments/20150219/0a602615/attachment-0003.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: icecast-2.5-somaxconn.patch Type: text/x-patch Size: 7524 bytes Desc: not available Url : http://lists.xiph.org/pipermail/icecast-dev/attachments/20150219/0a602615/attachment-0004.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: icecast-2.3.3-somaxconn.patch Type: text/x-patch Size: 4025 bytes Desc: not available Url : http://lists.xiph.org/pipermail/icecast-dev/attachments/20150219/0a602615/attachment-0005.bin
Hi, On 02/19/2015 03:40 PM, Stephan Leemburg wrote:> I don't know if you like top or bottom quoting. That seems to be a > big-little endian thing ;-) > > So, I will top quote and inline quote.playing it safe, ha!> Please see my comments inline, below. > > Kind regards, > Stephan > > > On 02/19/2015 04:18 PM, "Thomas B. R?cker" wrote: >> Hi, >> >> On 02/19/2015 03:07 PM, Stephan Leemburg wrote: >>> Hello Icecast-dev, >>> >>> I am new to this list. >> Welcome! > Thank you. > >>> I am working for the NPO, the Dutch Public Broadcasting agency. >>> We do a lot of icecast streaming. We run at least 20 icecast server >>> instances on our media streaming cluster. >> That's very nice to hear. >> >>> We ran into an issue that clients which where connecting to our streams >>> seemed to be 'hanging' on the connection setup frequently. The client >>> 'thinks' it is connected, but no data. >>> >>> People suggested that it probably had to do with the poll() call. So, I >>> looked into that. >>> >>> I found that the issue was actually caused by the very low listen >>> backlog (5). >>> >>> On our clusters, we typically set this to 8192. Yes it is high, but we >>> do a _lot_ of streaming and host very high volume websites. >> I'm not very familiar with socket programming, so will let Philipp >> comment on this. Interesting enough this issue hasn't come up so far as >> far as I can tell and there are some pretty high load deployments out there. > We often have 'bursts' of new connections. Due to something said on a > website, radio, tv, some top-2000 end of year event, etc. And when we > get a lot of simultanious connection requests, this becomes an issue. > > I wrote a small server to simulate it. And the linux backlog acts a > little bit different then expected.In the context of what you mention just below, yes, I'm immediately willing to believe that there is a problem. The scale also explains why it's probably not that commonly encountered.>> If you can share that info, what sort of concurrent listener load are we >> talking about? > Sure I can share, we are a national broadcasting agency funded by tax > money. So no secrets here ;-) > > I just asked the media streaming guys.. 70k icecast connections on a > regular day and 150k around special broadcasts (like top 2000 around new > year).That's rather sizable. There are two things I'd like to bring up in this context: * We'd be delighted if you/NPO could share a few things that you learned are important to deploy Icecast at that scale. Raising the ulimit is rather obvious, but you might have run into other things. * Also if there are more issues, we'd like to hear them, as we want to make Icecast even better. I personally believe in the simplicity of HTTP streaming. I've looked at HLS/DASH and there are various issues that make it unnecessarily hard to deal with for little value in return. Especially in web browsers the <audio> element starts to shape up and support simple HTTP streams rather well. The issue of supported codecs remains though, with Opus being a decent long run candidate. The main focus for Icecast in this context would be to help achieve better listener support on mobile devices, which traditionally seem to favour HLS. Large stations/networks just create an app that wraps things nicely, but there are many smaller ones. I see VLC as a good candidate in this context, but for that we also need to improve our stream directory at http://dir.xiph.org, as it currently only exports one big XML file. We're hoping to work out a good and flexible JSON API and help player-software projects integrate it. It's also part of our GSoC ideas.>>> Currently we are using icecast 2.3. We are migrating to 2.4. >>> So, I have written patches for 2.3 and 2.4, but also for the current 2.5 >>> git tree. >>> >>> Unfortunately, I am a newby when it comes to git (sorry). But I do have >>> unified diff patch files for the 2.3, 2.4 and 2.5 source trees. >>> >>> The patched 2.4 icecast was tested by our media streaming team and they >>> confirmed that their issue was solved by it. >>> >>> Can I submit them (and how)? >> Just send them as attachments to this list, or open a ticket over at >> https://trac.xiph.org. >> If trac is naughty and thinks you're a spammer, please let me know. >> >> If there are differences between them, then please the 2.5 and 2.4 >> patches. Otherwise 2.5 will do just fine. >> > I have attached the 2.4 and 2.5 patches.Thanks, Philipp took a look and we see the problem. Addressing this in a way that doesn't open us by default to DoS will be the main challenge. A likely outcome would be exposing the value in the configuration and only documenting it in the "very advanced" part. I'd expect that we'll address this one way or the other soon and it will be part of 2.5.0 later this year.>> Thanks a lot for taking the time to reach out to us! > Thank you for your Open Source contributions and efforts! > And as it is Open Source, we can fix problems ourself and share.As said we deeply appreciate that, it's what makes open source software better for everyone. Cheers Thomas
For HLS/DASH, I'm fairly certain SCPR tried running a chunking proxy in front of Icecast but found it not ideal; would be good to get their experiences in here. ? Micheil On 20 Feb 2015, at 11:45, "Thomas B. R?cker" <thomas at ruecker.fi> wrote:> Hi, > > On 02/19/2015 03:40 PM, Stephan Leemburg wrote: >> I don't know if you like top or bottom quoting. That seems to be a >> big-little endian thing ;-) >> >> So, I will top quote and inline quote. > > playing it safe, ha! > >> Please see my comments inline, below. >> >> Kind regards, >> Stephan >> >> >> On 02/19/2015 04:18 PM, "Thomas B. R?cker" wrote: >>> Hi, >>> >>> On 02/19/2015 03:07 PM, Stephan Leemburg wrote: >>>> Hello Icecast-dev, >>>> >>>> I am new to this list. >>> Welcome! >> Thank you. >> >>>> I am working for the NPO, the Dutch Public Broadcasting agency. >>>> We do a lot of icecast streaming. We run at least 20 icecast server >>>> instances on our media streaming cluster. >>> That's very nice to hear. >>> >>>> We ran into an issue that clients which where connecting to our streams >>>> seemed to be 'hanging' on the connection setup frequently. The client >>>> 'thinks' it is connected, but no data. >>>> >>>> People suggested that it probably had to do with the poll() call. So, I >>>> looked into that. >>>> >>>> I found that the issue was actually caused by the very low listen >>>> backlog (5). >>>> >>>> On our clusters, we typically set this to 8192. Yes it is high, but we >>>> do a _lot_ of streaming and host very high volume websites. >>> I'm not very familiar with socket programming, so will let Philipp >>> comment on this. Interesting enough this issue hasn't come up so far as >>> far as I can tell and there are some pretty high load deployments out there. >> We often have 'bursts' of new connections. Due to something said on a >> website, radio, tv, some top-2000 end of year event, etc. And when we >> get a lot of simultanious connection requests, this becomes an issue. >> >> I wrote a small server to simulate it. And the linux backlog acts a >> little bit different then expected. > > In the context of what you mention just below, yes, I'm immediately > willing to believe that there is a problem. The scale also explains why > it's probably not that commonly encountered. > > >>> If you can share that info, what sort of concurrent listener load are we >>> talking about? >> Sure I can share, we are a national broadcasting agency funded by tax >> money. So no secrets here ;-) >> >> I just asked the media streaming guys.. 70k icecast connections on a >> regular day and 150k around special broadcasts (like top 2000 around new >> year). > > That's rather sizable. > There are two things I'd like to bring up in this context: > > * We'd be delighted if you/NPO could share a few things that you learned > are important to deploy Icecast at that scale. Raising the ulimit is > rather obvious, but you might have run into other things. > > * Also if there are more issues, we'd like to hear them, as we want to > make Icecast even better. > > I personally believe in the simplicity of HTTP streaming. > I've looked at HLS/DASH and there are various issues that make it > unnecessarily hard to deal with for little value in return. > > Especially in web browsers the <audio> element starts to shape up and > support simple HTTP streams rather well. The issue of supported codecs > remains though, with Opus being a decent long run candidate. > > The main focus for Icecast in this context would be to help achieve > better listener support on mobile devices, which traditionally seem to > favour HLS. Large stations/networks just create an app that wraps things > nicely, but there are many smaller ones. I see VLC as a good candidate > in this context, but for that we also need to improve our stream > directory at http://dir.xiph.org, as it currently only exports one big > XML file. We're hoping to work out a good and flexible JSON API and help > player-software projects integrate it. It's also part of our GSoC ideas. > > >>>> Currently we are using icecast 2.3. We are migrating to 2.4. >>>> So, I have written patches for 2.3 and 2.4, but also for the current 2.5 >>>> git tree. >>>> >>>> Unfortunately, I am a newby when it comes to git (sorry). But I do have >>>> unified diff patch files for the 2.3, 2.4 and 2.5 source trees. >>>> >>>> The patched 2.4 icecast was tested by our media streaming team and they >>>> confirmed that their issue was solved by it. >>>> >>>> Can I submit them (and how)? >>> Just send them as attachments to this list, or open a ticket over at >>> https://trac.xiph.org. >>> If trac is naughty and thinks you're a spammer, please let me know. >>> >>> If there are differences between them, then please the 2.5 and 2.4 >>> patches. Otherwise 2.5 will do just fine. >> I have attached the 2.4 and 2.5 patches. > > Thanks, Philipp took a look and we see the problem. Addressing this in a > way that doesn't open us by default to DoS will be the main challenge. A > likely outcome would be exposing the value in the configuration and only > documenting it in the "very advanced" part. > > I'd expect that we'll address this one way or the other soon and it will > be part of 2.5.0 later this year. > > >>> Thanks a lot for taking the time to reach out to us! >> Thank you for your Open Source contributions and efforts! >> And as it is Open Source, we can fix problems ourself and share. > > As said we deeply appreciate that, it's what makes open source software > better for everyone. > > > Cheers > > Thomas > > _______________________________________________ > Icecast-dev mailing list > Icecast-dev at xiph.org > http://lists.xiph.org/mailman/listinfo/icecast-dev