Hi, I need some help. This question is not related to Xen but some one on this list may give me some idea so I am mailing. I have multiple video streaming servers running on some virtual machines running on Xen. For different subdomains. i.e. site1.mydomain.com site2.mydomain.com site3.mydomain.com site4.mydomain.com The front end to the world is apache2 on Dom0 To be able to reach the streaming server I embed a javascript in HTML pages as follows Code: <embed ..... var="rtmp://site1.my_domain.com">the problem is the website are many site1.mydomain.com site2.mydomain.com site3.mydomain.com site4.mydomain.com each on a separate virtual machine. Each of these four have their own streaming servers the front end to each of these four is apache running on Dom0. If I run rtmp on each of the subdomains (the virtual machines) at a different port how will I make sure a request such as rtmp://site1.mydomain.com rtmp://site2.mydomain.com goes to their respective servers. from the front end server from Dom0. What do I need to handle in this case ? IPTABLES came to mind instantly but from the client browser on internet when some one requests rtmp://site1.mydomain.com how will I make sure this rtmp request is mapped to a port different than 1935 as there are three other streaming servers which are also to respond to their respective requests ? For handling HTTP requests in this case I use Apache Reverse Proxy but for rtmp requests I am not clear as which direction to proceed. -- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jonathan Tripathy
2010-Aug-09 08:43 UTC
RE: [Xen-users] multiple streaming servers in a xen cloud
>The front end to the world is apache2 on Dom0This is the first thing you need to sort out. It''s a very bad idea to run customer facing things on a Dom0, especially things such as web pages with dynamic scripting code. The only thing that a Dom0 could be used for (apart from the normal Xen hosting stuff of course), is to be used to firewall between guests. Cheers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-09 08:55 UTC
Re: [Xen-users] multiple streaming servers in a xen cloud
So what should I use it for any more idea? Let me know I will try. On Mon, Aug 9, 2010 at 2:13 PM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:> >>The front end to the world is apache2 on Dom0 > > This is the first thing you need to sort out. It''s a very bad idea to run > customer facing things on a Dom0, especially things such as web pages with > dynamic scripting code. > > The only thing that a Dom0 could be used for (apart from the normal Xen > hosting stuff of course), is to be used to firewall between guests. > > Cheers_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher R. K.
2010-Aug-09 08:57 UTC
Re: [Xen-users] multiple streaming servers in a xen cloud
Hi there. Hrrrm, to "redirect" from rtmp to a different port/server, you could use a generic tcp proxy like HAProxy (i think) or tcpproxy or just use iptables with forwarding, or "balance" (http://balance.sourceforge.net/) or just use SSH to forward the traffic. Google for "linux tcp proxy" should give you a couple of results. And I agree with the other guy, you shouldn''t have critical, or user-sided applications running on a dom0, I think you should also virtualize it off to a own domU. Hope i was able to help.> Hi, > I need some help. > This question is not related to Xen but some one on this list may give > me some idea so I am mailing. > > > > I have multiple video streaming servers running on some virtual > machines running on Xen. > > For different subdomains. > i.e. > site1.mydomain.com > site2.mydomain.com > site3.mydomain.com > site4.mydomain.com > > > > The front end to the world is apache2 on Dom0 > > > > To be able to reach the streaming server I > embed a javascript in HTML pages > as follows > Code: > > <embed ..... > var="rtmp://site1.my_domain.com" > > > the problem is the website are many > > site1.mydomain.com > site2.mydomain.com > site3.mydomain.com > site4.mydomain.com > > each on a separate virtual machine. > > Each of these four have their own streaming servers the front end to > each of these four is apache running on Dom0. > > > If I run rtmp on each of the subdomains (the virtual machines) at a > different port > > how will I make sure > a request such as > > rtmp://site1.mydomain.com > rtmp://site2.mydomain.com > > goes to their respective servers. > from the front end server from Dom0. > > What do I need to handle in this case ? > > IPTABLES came to mind instantly but from the client browser on > internet when some one requests > rtmp://site1.mydomain.com > > how will I make sure this rtmp request is mapped to a port different > than > 1935 as there are three other streaming servers which are also to > respond to their respective requests > ? > > For handling HTTP requests in this case I use Apache Reverse Proxy > but for rtmp requests I am not clear as which direction to proceed. > > > -- > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-09 09:00 UTC
Re: [Xen-users] multiple streaming servers in a xen cloud
On Mon, Aug 9, 2010 at 2:27 PM, Christopher R. K. <feuerball_@gmx.net> wrote:> Hi there. > > Hrrrm, to "redirect" from rtmp to a different port/server, you could use a > generic tcp proxy like HAProxy (i think) or tcpproxy or just use iptables > with forwarding, or "balance" (http://balance.sourceforge.net/) or just use > SSH to forward the traffic. > Google for "linux tcp proxy" should give you a couple of results. > > And I agree with the other guy, you shouldn''t have critical, or user-sided > applications running on a dom0, I think you should also virtualize it off to > a own domU. > > > Hope i was able to help.Yes you were able to help. If some one else has some more ideas please drop in here. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher R. K.
2010-Aug-09 09:08 UTC
Re: [Xen-users] multiple streaming servers in a xen cloud
Excuse my terrible ascii-art, but... --------------------------- -------- ---------- | | RTMP |<-->| Apache | <------ Internet |(domU)| | (domU) | | -------- ---------- | -------- ^ ^ ^ | | RTMP |<--/ | | | |(domU)| / / | -------- / / | -------- / / | | RTMP |<--/ / | |(domU)| / | -------- / Dom0 | -------- / | | RTMP |<--/ | |(domU)| | -------- | --------------------------- I think something like that was what he had in mind, so that the dom0 only acts as "container" for the domUs and optionally the firewall. Am 09.08.2010 10:55, schrieb Tapas Mishra:> So what should I use it for any more idea? > Let me know I will try. > > On Mon, Aug 9, 2010 at 2:13 PM, Jonathan Tripathy<jonnyt@abpni.co.uk> wrote: >>> The front end to the world is apache2 on Dom0 >> This is the first thing you need to sort out. It''s a very bad idea to run >> customer facing things on a Dom0, especially things such as web pages with >> dynamic scripting code. >> >> The only thing that a Dom0 could be used for (apart from the normal Xen >> hosting stuff of course), is to be used to firewall between guests. >> >> Cheers > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-09 09:11 UTC
Re: [Xen-users] multiple streaming servers in a xen cloud
Ya right. On Mon, Aug 9, 2010 at 2:38 PM, Christopher R. K. <feuerball_@gmx.net> wrote:> Excuse my terrible ascii-art, but... > > --------------------------- > -------- ---------- | > | RTMP |<-->| Apache | <------ Internet > |(domU)| | (domU) | | > -------- ---------- | > -------- ^ ^ ^ | > | RTMP |<--/ | | | > |(domU)| / / | > -------- / / | > -------- / / | > | RTMP |<--/ / | > |(domU)| / | > -------- / Dom0 | > -------- / | > | RTMP |<--/ | > |(domU)| | > -------- | > --------------------------- > > I think something like that was what he had in mind, so that the dom0 only > acts as "container" for the domUs and optionally the firewall. > > Am 09.08.2010 10:55, schrieb Tapas Mishra: >> >> So what should I use it for any more idea? >> Let me know I will try. >> >> On Mon, Aug 9, 2010 at 2:13 PM, Jonathan Tripathy<jonnyt@abpni.co.uk> >> wrote: >>>> >>>> The front end to the world is apache2 on Dom0 >>> >>> This is the first thing you need to sort out. It''s a very bad idea to run >>> customer facing things on a Dom0, especially things such as web pages >>> with >>> dynamic scripting code. >>> >>> The only thing that a Dom0 could be used for (apart from the normal Xen >>> hosting stuff of course), is to be used to firewall between guests. >>> >>> Cheers >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Tapas http://mightydreams.blogspot.com http://wiki.xensource.com/xenwiki/Xen_on_4_app_servers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Simon Billis
2010-Aug-09 09:35 UTC
RE: [Xen-users] multiple streaming servers in a xen cloud
Hi,> I have multiple video streaming servers running on some virtual > machines running on Xen. > > For different subdomains. > i.e. > site1.mydomain.com > site2.mydomain.com > site3.mydomain.com > site4.mydomain.comI assume that these different sites are streaming different content and do not need to be load balanced or HA.> The front end to the world is apache2 on Dom0This is not recommended, you would be better off creating another DomU and having apache in there.> To be able to reach the streaming server I > embed a javascript in HTML pages > as follows > Code: > > <embed ..... > var="rtmp://site1.my_domain.com" > > the problem is the website are many > > site1.mydomain.com > site2.mydomain.com > site3.mydomain.com > site4.mydomain.com > > each on a separate virtual machine.This is the important bit - If each machine is a different DomU and you have setup the networking in such a fashion that each DomU has its own IP address then nothing has to be done to the url to make this work. You don''t even have to change the listening port on the streaming servers.> Each of these four have their own streaming servers the front end to > each of these four is apache running on Dom0.This is one instance of apache with either a number of virtual hosts or a single host with a, for example, single page with the different urls on it.> > If I run rtmp on each of the subdomains (the virtual machines) at a > different port > > how will I make sure > a request such as > > rtmp://site1.mydomain.com > rtmp://site2.mydomain.com > > goes to their respective servers. > from the front end server from Dom0.The simplest thing to do is either ensure that the DomU''s containing the streaming servers are on different IP addresses and then you can use those urls for the specific DomU without change, or if you really want to have different ports then specify the port in the url such as: rtmp://site1.mydomain.com:8888 where 8888 is the port that the streaming server is listening on.> What do I need to handle in this case ? > > IPTABLES came to mind instantly but from the client browser on > internet when some one requests > rtmp://site1.mydomain.com > > how will I make sure this rtmp request is mapped to a port different > than > 1935 as there are three other streaming servers which are also to > respond to their respective requests > ? > > For handling HTTP requests in this case I use Apache Reverse Proxy > but for rtmp requests I am not clear as which direction to proceed. >In general, try to keep it simple, the above can work, but is difficult to implement and maintain. It requires an good knowledge and understanding of iptables and Squid in each case. If you provide some specific information on the networking side of the DomU''s then I might be able to be more specific. Rgds Simon. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-09 09:48 UTC
Re: [Xen-users] multiple streaming servers in a xen cloud
Simons thanks for detailed reply I here is the information you requested quotes from your messages On Mon, Aug 9, 2010 at 3:05 PM, Simon Billis <simon@houxou.com> wrote:> Hi, >> I have multiple video streaming servers running on some virtual >> machines running on Xen. >> >> For different subdomains. >> i.e. >> site1.mydomain.com >> site2.mydomain.com >> site3.mydomain.com >> site4.mydomain.com > > I assume that these different sites are streaming different content and do > not need to be load balanced or HA.Yes very correct.>> The front end to the world is apache2 on Dom0 > > This is not recommended, you would be better off creating another DomU and > having apache in there.I am not clear with this point. The Dom0 has public IP so you mean to say I create another guest which would have my public IP.>> To be able to reach the streaming server I >> embed a javascript in HTML pages >> as follows >> Code: >> >> <embed ..... >> var="rtmp://site1.my_domain.com" >> >> the problem is the website are many >> >> site1.mydomain.com >> site2.mydomain.com >> site3.mydomain.com >> site4.mydomain.com >> >> each on a separate virtual machine. > > This is the important bit - If each machine is a different DomU and you have > setup the networking in such a fashion that each DomU has its own IP address > then nothing has to be done to the url to make this work >. You don''t even > have to change the listening port on the streaming servers.Each DomU has its own IP but these are known to machines on my LAN. Meaning in a subnet of Class C 192.168.1.2 to 192.168.1.5 are DomU IPs and Dom0 IP is 192.168.1.1 also Dom0 has a public IP.>> Each of these four have their own streaming servers the front end to >> each of these four is apache running on Dom0. > > This is one instance of apache with either a number of virtual hosts or a > single host with a, for example, single page with the different urls on it. >Each Virtual Machine has an Apache vhost on it. And on Dom0 I have separate vhosts running which use Reverse Proxy to reach the virtual hosts on guest machines. i.e. for DomU 1 at Dom0 the vhost is <VirtualHost *:80> ServerName site1.mydomain.com ProxyPass / http://domU1 ProxyPassReverse / http://domU1 </VirtualHost > and the virtual host at DomU 1 is <VirtualHost *:80> ServerName site1.mydomain.com DocumentRoot /var/www ..... ...rest of the stuff </VirtualHost >>> If I run rtmp on each of the subdomains (the virtual machines) at a >> different port >> >> how will I make sure >> a request such as >> >> rtmp://site1.mydomain.com >> rtmp://site2.mydomain.com >> >> goes to their respective servers. >> from the front end server from Dom0.For some time leave this part of question let us run all the rtmp server on 1935 only.> The simplest thing to do is either ensure that the DomU''s containing the > streaming servers are on different IP addressesI am using different IP (but they are only available on LAN) to internet the end user does not even know that he got served via Different Virtual Machines.> and then you can use those > urls for the specific DomU without change, or if you really want to have > different ports then specify the port in the url such as: > rtmp://site1.mydomain.com:8888 where 8888 is the port that the streaming > server is listening on.Not very clear about this one.> >> What do I need to handle in this case ? >> >> IPTABLES came to mind instantly but from the client browser on >> internet when some one requests >> rtmp://site1.mydomain.com >> >> how will I make sure this rtmp request is mapped to a port different >> than >> 1935 as there are three other streaming servers which are also to >> respond to their respective requestsWell I want the requests coming at 1935 for rtmp protocol based on host names be served via different virtual machines.(which have different domains to server)>> For handling HTTP requests in this case I use Apache Reverse Proxy >> but for rtmp requests I am not clear as which direction to proceed. >> > > In general, try to keep it simple, the above can work, but is difficult to > implement and maintain. It requires an good knowledge and understanding of > iptables and Squid in each case.Well all this is for a learning experience so let me know, I know how to set up IPTABLE and Squid. What else is needed.> If you provide some specific information on the networking side of the > DomU''s then I might be able to be more specific.DomU1 192.168.1.2 DomU2 192.168.1.3 DomU3 192.168.1.4 DomU4 192.168.1.5 Dom0 is 192.168.1.1> Rgds > > Simon. > > >Let me know if you need any more information. -- Tapas http://mightydreams.blogspot.com http://wiki.xensource.com/xenwiki/Xen_on_4_app_servers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Simon Billis
2010-Aug-09 10:44 UTC
RE: [Xen-users] multiple streaming servers in a xen cloud
Hi,> >> The front end to the world is apache2 on Dom0 > > > > This is not recommended, you would be better off creating another > DomU and > > having apache in there. > I am not clear with this point. > The Dom0 has public IP so you mean to say I create another guest which > would have my public IP.I''m not sure about this - I don''t run xen in this configuration, I split the firewalling off to dedicated hardware and then have my Xen host behind the firewall doing stuff however, here is a stab at it - I''m sure that someone here will correct me :-)>From the info on your wiki page I would do the following:Create an instance called "e" with an IP address of 192.168.0.15 and have apache on it with the config that you have on Dom0 (G). I would then use the firewall on "G" to forward requests from the Public IP to "e" to be served by the apache instance there. The Apache on "e" is doing the reverse proxy stuff and all the connections to the other servers is handled by that, so all should just work. This fixes the first issue with running services on Dom0 Info from you wiki page: IP of Domu1:192.168.0.11 Lets call it as a IP of Domu2:192.168.0.12 Lets call it as b IP of Domu3:192.168.0.13 Lets Call it as c IP of Domu4:192.168.0.14 Lets Call it as d Ip of Dom0 :192.168.0.100 (Will behave as Gateway for DomU''s) Lets call Dom0 as A Gateway for the network is 192.168.0.1 We will call it as G.> >> Each of these four have their own streaming servers the front end to > >> each of these four is apache running on Dom0. > >> rtmp://site1.mydomain.com > >> rtmp://site2.mydomain.com > >> > >> goes to their respective servers. > >> from the front end server from Dom0. > > For some time leave this part of question let us run all the rtmp > server on 1935 only. >Ahhahhh! Got it... You are attempting to have site1.mydomain streaming server and site2.mydomain streaming server on the same single Public Ip Address. OK - This is not simple to do with a single Public IP address. It requires either port redirection or layer7 firewalls/reverse proxies such as SOCKS or perhaps Squid. Port redirection is handled on the firewall using iptables. It can only work if you are able to control the url that is being presented to the outside world. E.g. in normal web browsers you can specify alternate ports for the url in this manner: http://site1.mydomain:8080 this tells the browser to connect to port 8080 on the ip address that site1.mydomain points to. This doesn''t work if you have no control over the url or are expecting web crawlers etc to access the site. However, in your case I "think" that you have some control over the url that is presented to the outside world, so there for you are able to do the following: rtmp://site1.mydomain:8888 ---> your public IP address port 8888 which then hits the firewall and is redirected: publicIP port 8888 -----> privateip 192.168.0.1 port1935 rtmp://site2.mydomain:9999 ---> your public IP address port 9999 which then hits the firewall and is redirected: publicIP port 9999 -----> privateip 192.168.0.2 port1935 Simple to do and works well. Layer 7 proxies / firewall have to look at the url requested and then determine which server should answer the query. This is a question that you would need to ask on the relevant list for the chosen proxy. HTH Simon. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-09 11:48 UTC
Re: [Xen-users] multiple streaming servers in a xen cloud
Simon thanks for the answer. I am clear with what to look for thanks once again.> rtmp://site1.mydomain:8888 ---> your public IP address port 8888 which then > hits the firewall and is redirected: > publicIP port 8888 -----> privateip 192.168.0.1 port1935 > > rtmp://site2.mydomain:9999 ---> your public IP address port 9999 which then > hits the firewall and is redirected: > > publicIP port 9999 -----> privateip 192.168.0.2 port1935 > > > Simple to do and works well.Got it was so confused among various streaming servers that had missed this part. Is seriously easy enough but I would not like to do this one. I will go with the following suggestion of yours.> Layer 7 proxies / firewall have to look at the url requested and then > determine which server should answer the query. This is a question that you > would need to ask on the relevant list for the chosen proxy. >This is what I need to be looking for. In case you have some more idea on this one let me know. Thanks once again. -- Tapas http://mightydreams.blogspot.com http://wiki.xensource.com/xenwiki/Xen_on_4_app_servers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-10 04:41 UTC
[SPAM] Re: [Xen-users] multiple streaming servers in a xen cloud
Since I had started this thread here I came across http://l7-filter.clearfoundation.com/docs/readme if some one comes across this thread by chance this should help them. -- Tapas http://mightydreams.blogspot.com http://wiki.xensource.com/xenwiki/Xen_on_4_app_servers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
francisco javier funes nieto
2010-Aug-10 09:15 UTC
Re: [SPAM] Re: [Xen-users] multiple streaming servers in a xen cloud
This is full off-topic, but I use Mikrotik RouterOS (1) with L7 capable Firewall for this kind of problems... And you could use it with Xen too. (1) http://wiki.mikrotik.com/wiki/Manual:IP/Firewall/L7 2010/8/10 Tapas Mishra <mightydreams@gmail.com>> Since I had started this thread here > I came across > http://l7-filter.clearfoundation.com/docs/readme > if some one comes across this thread by chance this should help them. > > -- > Tapas > http://mightydreams.blogspot.com > http://wiki.xensource.com/xenwiki/Xen_on_4_app_servers > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- _____________________________________________ Francisco Javier Funes Nieto [esencia@gmail.com] CANONIGOS Servicios Informáticos para PYMES. Cl. Cruz 2, 1º Oficina 7 Tlf: 958.536759 / 661134556 Fax: 958.521354 GRANADA - 18002 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-10 11:00 UTC
Re: [SPAM] Re: [Xen-users] multiple streaming servers in a xen cloud
Although I do not have this router thanks for this information. Will see if possible. -- Tapas http://mightydreams.blogspot.com http://wiki.xensource.com/xenwiki/Xen_on_4_app_servers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tapas Mishra
2010-Aug-10 11:19 UTC
Re: [SPAM] Re: [Xen-users] multiple streaming servers in a xen cloud
I got this suggestion also from some one. Since I had asked this question here so posting this message. Pretty much nothing using "standard" approach. Because the target server/application is deeply embedded into the RTMP traffic of that particular connection, you have to unpack it to know the target server/application. For this, you need to accept the connection in the front server. Once you did that, the only way to keep it functioning is to forward all future traffic/requests to the chosen server while keeping the initial connection alive. But, as I said, you will have 2 connections that you MUST maintain: 1. flash_player --> front_server 2. front_server --> target_lan_server To make the problem even complex than it already is, the surrogate front_server MUST do the correct RTMP handshake. Otherwise you will not be able to play h264 content. And the sad part is that the information required (servername/application) only occurs on the connection AFTER the handshake is completed (in connection number 1) Bottom line, very hard. However, perfectly doable with rtmpd. But you have to hack the server pretty drastically to achieve that. In any case, the big problem persists. You will have a single point of failure: front_server. To make the matter even worst, the major bottleneck will be the bandwidth. ALL traffic will go through one server:front_server. Unless you have multiple IPs/NetworkCards/InternetPipes in the front_server, _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users