Icecast - Oct 2014 - Running Icecast2 behind Amazon Elastic Load Balancer (ELB)

Hi Thomas,

We need a load balancer to allow for a single URL "entry point" to
multiple
Icecast servers - we intend to host multiple Icecast servers behind the
load balancer so that we can expand / contract capacity as required - we
would add servers into the load balancer pool. This also allows us to scale
beyond the number of listeners a single Icecast server can support, while
still maintaining the "illusion" of a single streaming server.

It also provides redundancy / failover capability by allowing us to balance
connections across Icecast servers in multiple AWS availability zones.

I'm aware of some "playlist based" loadbalancing techniques using
.m3u or
HTTP redirects to the icecast servers, but this is not ideal for us, as a)
we need to support some legacy players (eg. set top boxes) that don't
support redirects or m3u's, and b) using ELB allows us to very easily
provision backend health checks, multi availability zone redundancy, and
auto scaling of Icecast servers.

Hope that answers your question,

Cheers,

Matt



On 14 October 2014 13:41, "Thomas B. R?cker" <thomas at
ruecker.fi> wrote:
> Hi Matt,
>
> On 10/13/2014 07:08 PM, Matt Gray wrote:
> > Hello,
> >
> > I'm writing to see if anyone here has had any success running
Icecast2
> > (specifically I am using 2.4) on Amazon EC2 behind an Elastic Load
> > Balancer?
>
> Why do you think you need a load balancer in the first place?
>
> Icecast saturates the network before you notice any other significant
> resource usage.
>
> I'm happy to help you with your problem if there is a good reason, but
> would like to first make sure that this needs to be addressed.
>
>
> Cheers
>
> Thomas
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>


-- 
*Matt Gray*
System Administrator

*T:* +44(0)207 099 7777
*E:* matt.gray at 7digital.com
*W:* www.7digital.com
<http://www.google.com/url?q=http%3A%2F%2Fwww.7digital.com&sa=D&sntz=1&usg=AFrqEzeLlO75mVeA-8Yi-niLjEpFokXaYA>

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.xiph.org/pipermail/icecast/attachments/20141014/ae639d1a/attachment.htm

On 10/14/2014 12:56 PM, Matt Gray wrote:
> Hi Thomas,
>
> We need a load balancer to allow for a single URL "entry point"
to
> multiple Icecast servers - we intend to host multiple Icecast servers
> behind the load balancer so that we can expand / contract capacity as
> required - we would add servers into the load balancer pool. This also
> allows us to scale beyond the number of listeners a single Icecast
> server can support, while still maintaining the "illusion" of a
single
> streaming server.
That is only really relevant if you expect to saturate the upstream
bandwidth of the instance running Icecast. Which nowadays is in the
10-20k listener range at the very least. Clustering through DNS also
removes that limitation.
For context, such listener numbers are only achieved by the largest
on-line radio stations.
Does an ELB have more upstream bandwidth than a single instance? Would
be one question. If not (iow >>1Gbit) you need a different approach
anyway.

> It also provides redundancy / failover capability by allowing us to
> balance connections across Icecast servers in multiple AWS
> availability zones.
Redundancy/failover might be worth looking at. I'm not exactly sure if
adding another SPOF is the right approach, but thankfully it's not my
decision.
Out of curiosity, is ELB able to span several AZ?

Anyway, Icecast on the listener side is plain HTTP, I'd try that first.
I'm not sure how well it will cope with the extremely long lived
connections though. In my experience adding the complexity of a reverse
proxy decreases Icecast availability and reliability rather than improve it.

> I'm aware of some "playlist based" loadbalancing techniques
using .m3u
> or HTTP redirects to the icecast servers, but this is not ideal for
> us, as a) we need to support some legacy players (eg. set top boxes)
> that don't support redirects or m3u's, and
I'd not go for a playlist as it relies on clients. I'd probably just use
a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
distribution. One stable and fully transparent URL. That's also easier
to implement for reliable and robust availability in my opinion. Turns
out Amazon even offers it as a service: Amazon Route 53.

> b) using ELB allows us to very easily provision backend health checks,
> multi availability zone redundancy, and auto scaling of Icecast servers.
OK, that sentence ticks a row on my bingo card. ;-)
You're set on using ELB for some reason and I won't argue further with
it. I'll just warn that it's probably not ideal in both HTTP and TCP
mode. You should do significant testing to ensure things work as they do
without it in the way.
Why it doesn't close the TCP connection after the original connection is
gone is not clear to me. It might be a bug or a feature of ELB. You
should actually check AWS documentation or contact Amazon support about
that. You can chose to ignore it and the system will eventually kill
them, but you should make sure to have enough of head room for
connections available on the Icecast machine (ulimit and server config
limits).

I know I sound quite defensive about this topic, but the point I'm
trying to make is, that it adds not well known behaviour, as it's not
fully transparent, and constitutes another single point of failure. Also
there are suitable other options. (I'd even make the same point for a
plain web server)


Cheers

Thomas

On 10/14/2014 12:56 PM, Matt Gray wrote:
> Hi Thomas,
>
> We need a load balancer to allow for a single URL "entry point"
to
> multiple Icecast servers - we intend to host multiple Icecast servers
> behind the load balancer so that we can expand / contract capacity as
> required - we would add servers into the load balancer pool. This also
> allows us to scale beyond the number of listeners a single Icecast
> server can support, while still maintaining the "illusion" of a
single
> streaming server.
That is only really relevant if you expect to saturate the upstream
bandwidth of the instance running Icecast. Which nowadays is in the
10-20k listener range at the very least. Clustering through DNS also
removes that limitation.
For context, such listener numbers are only achieved by the largest
on-line radio stations.
Does an ELB have more upstream bandwidth than a single instance? Would
be one question. If not (iow >>1Gbit) you need a different approach
anyway.

> It also provides redundancy / failover capability by allowing us to
> balance connections across Icecast servers in multiple AWS
> availability zones.
Redundancy/failover might be worth looking at. I'm not exactly sure if
adding another SPOF is the right approach, but thankfully it's not my
decision.
Out of curiosity, is ELB able to span several AZ?

Anyway, Icecast on the listener side is plain HTTP, I'd try that first.
I'm not sure how well it will cope with the extremely long lived
connections though. In my experience adding the complexity of a reverse
proxy decreases Icecast availability and reliability rather than improve it.

> I'm aware of some "playlist based" loadbalancing techniques
using .m3u
> or HTTP redirects to the icecast servers, but this is not ideal for
> us, as a) we need to support some legacy players (eg. set top boxes)
> that don't support redirects or m3u's, and
I'd not go for a playlist as it relies on clients. I'd probably just use
a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
distribution. One stable and fully transparent URL. That's also easier
to implement for reliable and robust availability in my opinion. Turns
out Amazon even offers it as a service: Amazon Route 53.

> b) using ELB allows us to very easily provision backend health checks,
> multi availability zone redundancy, and auto scaling of Icecast servers.
OK, that sentence ticks a row on my bingo card. ;-)
You're set on using ELB for some reason and I won't argue further with
it. I'll just warn that it's probably not ideal in both HTTP and TCP
mode. You should do significant testing to ensure things work as they do
without it in the way.
Why it doesn't close the TCP connection after the original connection is
gone is not clear to me. It might be a bug or a feature of ELB. You
should actually check AWS documentation or contact Amazon support about
that. You can chose to ignore it and the system will eventually kill
them, but you should make sure to have enough of head room for
connections available on the Icecast machine (ulimit and server config
limits).

I know I sound quite defensive about this topic, but the point I'm
trying to make is, that it adds not well known behaviour, as it's not
fully transparent, and constitutes another single point of failure. Also
there are suitable other options. (I'd even make the same point for a
plain web server)


Cheers

Thomas

On 14 October 2014 15:01, "Thomas B. R?cker" <thomas at
ruecker.fi> wrote:
> On 10/14/2014 12:56 PM, Matt Gray wrote:
> > Hi Thomas,
> >
> > We need a load balancer to allow for a single URL "entry
point" to
> > multiple Icecast servers - we intend to host multiple Icecast servers
> > behind the load balancer so that we can expand / contract capacity as
> > required - we would add servers into the load balancer pool. This also
> > allows us to scale beyond the number of listeners a single Icecast
> > server can support, while still maintaining the "illusion"
of a single
> > streaming server.
>
> That is only really relevant if you expect to saturate the upstream
> bandwidth of the instance running Icecast. Which nowadays is in the
> 10-20k listener range at the very least. Clustering through DNS also
> removes that limitation.
>
Unfortunately this is not the case in EC2, as the upstream bandwidth is not
as high as one would expect hosting one's own server (the AWS decision is
out of my hands unfortunately). With an m3.medium we are able to support
about 2000 listeners per server.

> For context, such listener numbers are only achieved by the largest
> on-line radio stations.
> Does an ELB have more upstream bandwidth than a single instance? Would
> be one question. If not (iow >>1Gbit) you need a different approach
anyway.
>
ELB has much greater upstream bandwidth than an of the amazon instance
sizes. It's effectively unlimited, it grows as connections increase.

> > It also provides redundancy / failover capability by allowing us to
> > balance connections across Icecast servers in multiple AWS
> > availability zones.
>
> Redundancy/failover might be worth looking at. I'm not exactly sure if
> adding another SPOF is the right approach, but thankfully it's not my
> decision.
> Out of curiosity, is ELB able to span several AZ?
>
ELB is not a SPOF (at least an entire Amazon region has to go down for an
ELB to completely fail), because it scales to cope with increased
connections. ELB is able to span AZs within a region.

Anyway, Icecast on the listener side is plain HTTP, I'd try that
first.
> I'm not sure how well it will cope with the extremely long lived
> connections though. In my experience adding the complexity of a reverse
> proxy decreases Icecast availability and reliability rather than improve
> it.
>
>
> > I'm aware of some "playlist based" loadbalancing
techniques using .m3u
> > or HTTP redirects to the icecast servers, but this is not ideal for
> > us, as a) we need to support some legacy players (eg. set top boxes)
> > that don't support redirects or m3u's, and
>
> I'd not go for a playlist as it relies on clients. I'd probably
just use
> a RR-DNS with very short TTL instead. Maybe geo-DNS if I care about
> distribution. One stable and fully transparent URL. That's also easier
> to implement for reliable and robust availability in my opinion. Turns
> out Amazon even offers it as a service: Amazon Route 53.
>
Yes we've looked at Route 53, perhaps this is an option we will go with.
> b) using ELB allows us to very easily provision backend health checks,
> > multi availability zone redundancy, and auto scaling of Icecast
servers.
>
> OK, that sentence ticks a row on my bingo card. ;-)
> You're set on using ELB for some reason and I won't argue further
with
> it.

We are certainly not set on it, because of the issues that I've explained
(which might well be to do with ELB) I'm just trying to to understand the
behaviour of Icecast in that environment.

> I'll just warn that it's probably not ideal in both HTTP and TCP
> mode. You should do significant testing to ensure things work as they do
> without it in the way.
> Why it doesn't close the TCP connection after the original connection
is
> gone is not clear to me. It might be a bug or a feature of ELB. You
> should actually check AWS documentation or contact Amazon support about
> that.

I have also taken this up with AWS support and their stance is that sockets
in CLOSE_WAIT are waiting for the backend server to close them, and there
is nothing that the ELB can do about that.

> You can chose to ignore it and the system will eventually kill
> them, but you should make sure to have enough of head room for
> connections available on the Icecast machine (ulimit and server config
> limits).
>
Unfortunately we can't, because of the issues you mention and also we need
to use the disconnection event to log listener usage etc.

> I know I sound quite defensive about this topic, but the point I'm
> trying to make is, that it adds not well known behaviour, as it's not
> fully transparent, and constitutes another single point of failure. Also
> there are suitable other options. (I'd even make the same point for a
> plain web server)

 You've every right to be defensive :) I'm just trying to understand
what
causes the CLOSE_WAIT states. You are right, there are plenty of other
options.

In general, what are people using to provide a high availability icecast
service?

Thanks for your time putting together such a comprehensive response.

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.xiph.org/pipermail/icecast/attachments/20141014/ab2fbbb2/attachment.htm