I need to installa a streaming server for live videos and for video on demand. My choice is red5. Do you think I can use a xen virtual machine to run a streaming service? Or it is better not to virtualize? Sorry for my bad english. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rainer Fuegenstein
2009-Sep-18 10:21 UTC
Re: [Xen-users] streaming server on a virtual machine.
can''t say much about video streaming, but I''m running a shoutcast audio stream relay in a centos5 domU (dom0 is also centos5) and it is running fine with about 50 streams in parallel. but your requirements may be higher. hth Mauro wrote:> I need to installa a streaming server for live videos and for video on demand. > My choice is red5. > Do you think I can use a xen virtual machine to run a streaming service? > Or it is better not to virtualize? > Sorry for my bad english. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I don''t see why not. If you''re concerned about latency you could always pin the server to a physical CPU. (Lately we''ve been doing a lot of physical-to-virtual migrations, and I''ve found myself asking: "What kinds of servers do we have that should never be virtual?" There aren''t many on that list.)> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Mauro > Sent: Friday, September 18, 2009 6:08 AM > To: xen-users > Subject: [Xen-users] streaming server on a virtual machine. > > I need to installa a streaming server for live videos and for video ondemand.> My choice is red5. > Do you think I can use a xen virtual machine to run a streamingservice?> Or it is better not to virtualize? > Sorry for my bad english. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>:> I don''t see why not. If you''re concerned about latency you could always > pin the server to a physical CPU.I''m thinking about the limitation of the NIC. Sorry for my bad english I''m italian and I don''t know how to express my concepts. In the same phisical machine I''ve configured a domU with a mail server. The network traffic in and out of this server mail is not little. I''ve others nic in the same machine perhaps I need to assign a different nic to the virtual streaming server? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Carsten Schiers
2009-Sep-18 12:40 UTC
AW: Re: [Xen-users] streaming server on a virtual machine.
I have a virtual server running with 8 DomUs in addition to the Dom0 of course. one is a mail server, one is a firewall, one in DMZ as webserver, one is a file Server /SAMBA, NFS), etc. I have two PVR (VDR, a video tape recoder). One of them, I have given a physical NIC with PCI passthrough, the other not. In addition, I have set up a streaming server that serves HD (1080p), also though bridging. There is no issue with all of them, so if there is a spare NIC available, I think I would use it. Otherwise, I see no real issue with network load. You can also use cred and weight to prioritize your streaming domain, just in case There is heavy load. See http://book.xen.prgmr.com/mediawiki/index.php/Scheduling Pleas note that I also use cpufreq-mangement, so my CPU ist normaly at 2x1GHz, and even streaming will not change a lot on that. Best Regards, Carsten. -----Ursprüngliche Nachricht----- Von: Mauro [mailto:mrsanna1@gmail.com] Gesendet: Freitag, 18. September 2009 12:52 An: Jeff Sturm Cc: xen-users Betreff: Re: [Xen-users] streaming server on a virtual machine. 2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>:> I don''t see why not. If you''re concerned about latency you couldalways> pin the server to a physical CPU.I''m thinking about the limitation of the NIC. Sorry for my bad english I''m italian and I don''t know how to express my concepts. In the same phisical machine I''ve configured a domU with a mail server. The network traffic in and out of this server mail is not little. I''ve others nic in the same machine perhaps I need to assign a different nic to the virtual streaming server? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: Mauro [mailto:mrsanna1@gmail.com] > Sent: Friday, September 18, 2009 6:52 AM > To: Jeff Sturm > Cc: xen-users > Subject: Re: [Xen-users] streaming server on a virtual machine. > > 2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>: > > I don''t see why not. If you''re concerned about latency you could always > > pin the server to a physical CPU. > > I''m thinking about the limitation of the NIC.I''ve pushed xennet close to wire speeds from a domU. The paravirt drivers do not seem to present much of a bottleneck. I''d encourage anyone asking similar questions to try it for themselves... I did, and learned a lot about Xen by pushing it to its limits. One thing to note however. Xen''s network layer may not cope well with a substantial number (10-20) of virtual hosts performing high rates of concurrent network I/O. I do not know if this is a limitation of the net back driver, Linux bridging or something else...I spotted it early on when I had each domU mounting filesystems from a networked SAN. Switching the SAN traffic over to dom0 and exporting block devices to each domU made a world of difference, and allowed us to use CLVM to easily manage the storage as well. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>:> I don''t see why not. If you''re concerned about latency you could always > pin the server to a physical CPU. > > (Lately we''ve been doing a lot of physical-to-virtual migrations, and > I''ve found myself asking: "What kinds of servers do we have that should > never be virtual?" There aren''t many on that list.) >You say that a streaming server can be installed on a virtual machine. I don''t want to create any kind of disturb but I''ve asked the same question to the KVM mailing list and they said that a streaming server (I user red5), due to the high I/O traffic, it must absolutely not virtualized. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Sep-26 10:05 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sat, Sep 26, 2009 at 4:42 PM, Mauro <mrsanna1@gmail.com> wrote:> 2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>: >> I don''t see why not. If you''re concerned about latency you could always >> pin the server to a physical CPU. >> >> (Lately we''ve been doing a lot of physical-to-virtual migrations, and >> I''ve found myself asking: "What kinds of servers do we have that should >> never be virtual?" There aren''t many on that list.) >> > You say that a streaming server can be installed on a virtual machine. > I don''t want to create any kind of disturb but I''ve asked the same > question to the KVM mailing list and they said that a streaming server > (I user red5), due to the high I/O traffic, it must absolutely not > virtualized.Here''s my general point of view of that : one of the purposes of virtualization is to consolidate servers with low utilization to increase system utilization and reduce hardware requirements, thus reducing cost. Here''s an over-simplified example: lets say your physical server is capable of providing 2 GHz CPU and 100 Mbps network bandwidth, and your streaming server will require 500 MHz CPU and 90 Mbps network. Disk I/O is not an issue since all data is cached in memory (again, this is just an example). With that setup and requirements, it''s not a good idea to virtualize the streaming server as it already have high network utilization. The case is different if you have 1 Gbps NIC, or if you have an extra 100 Mbps NIC that you can dedicate to domU (either with dedicated bridge or PCI passthrough). The limiting factor migh differ depending on usage. It could be CPU, memory, disk (throughput and IOPS), or network. It could even be latency (like with VOIP). In any case, it''s best to determine what your requirements are and what you''re physical server is capable of before deciding whether or not to go down virtualization route. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
We''re running a virtualized streaming server without any problems. Our server provides flash, windows media, real, quicktime and shoutcast platforms without any hickups so far. Naturally the feasibility of virtualizing a streaming server depends on how heavy load it is put under; in our case this server has a sustained user load of 30Mbps based on traffic and it doesn''t generate really any load at all on the server hardware yet. Also no haven''t seen any performance issues with network or disk IO. There are some half a dozen other domU''s running on the same system. The flash streams are provided via Red5, but that''s just one of the services the system is offering, so I''d say virtualization of a media server is fine, but naturally your mileage may vary; if your hardware server is running close to its limits (be it IO or CPU), then virtualization (to the same level of hardware) wouldn''t make any sense as mentioned earlier. cheers, Antti Kanes> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Mauro > Sent: 26. syyskuuta 2009 12:43 > To: Jeff Sturm > Cc: xen-users > Subject: Re: [Xen-users] streaming server on a virtual machine. > > 2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>: > > I don''t see why not. If you''re concerned about latency you could > > always pin the server to a physical CPU. > > > > (Lately we''ve been doing a lot of physical-to-virtual > migrations, and > > I''ve found myself asking: "What kinds of servers do we have that > > should never be virtual?" There aren''t many on that list.) > > > You say that a streaming server can be installed on a virtual machine. > I don''t want to create any kind of disturb but I''ve asked the > same question to the KVM mailing list and they said that a > streaming server (I user red5), due to the high I/O traffic, > it must absolutely not virtualized. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/9/26 Antti Kanes <antti@kanes.fi>:> > We''re running a virtualized streaming server without any problems. > Our server provides flash, windows media, real, quicktime and shoutcast > platforms without any hickups so far. > > Naturally the feasibility of virtualizing a streaming server depends on how > heavy load it is put under; in our case this server has a sustained user > load of 30Mbps based on traffic and it doesn''t generate really any load at > all on the server hardware yet. Also no haven''t seen any performance issues > with network or disk IO. > > There are some half a dozen other domU''s running on the same system.The traffic of all domU go over the same network card? Or you have assigned different NIC to some domU? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Same 1Gbps NIC for all the external traffic. No problems there. But as I said, the media services usually run at 30MBps, so it''s hardly an issue, the server is connected to the internet with just some 100Mbps bandwidth, so it''s easily handled by the system. Another NIC is dedicated for traffic between the servers (there''s an internal network with the domU''s and a couple of hardware servers).> -----Original Message----- > From: Mauro [mailto:mrsanna1@gmail.com] > Sent: 26. syyskuuta 2009 16:23 > To: Antti Kanes > Cc: Jeff Sturm; xen-users > Subject: Re: [Xen-users] streaming server on a virtual machine. > > 2009/9/26 Antti Kanes <antti@kanes.fi>: > > > > We''re running a virtualized streaming server without any problems. > > Our server provides flash, windows media, real, quicktime and > > shoutcast platforms without any hickups so far. > > > > Naturally the feasibility of virtualizing a streaming > server depends > > on how heavy load it is put under; in our case this server has a > > sustained user load of 30Mbps based on traffic and it > doesn''t generate > > really any load at all on the server hardware yet. Also no haven''t > > seen any performance issues with network or disk IO. > > > > There are some half a dozen other domU''s running on the same system. > > The traffic of all domU go over the same network card? > Or you have assigned different NIC to some domU? >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-26 14:05 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sat, Sep 26, 2009 at 2:42 AM, Mauro <mrsanna1@gmail.com> wrote:> 2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>: > > I don''t see why not. If you''re concerned about latency you could always > > pin the server to a physical CPU. > > > > (Lately we''ve been doing a lot of physical-to-virtual migrations, and > > I''ve found myself asking: "What kinds of servers do we have that should > > never be virtual?" There aren''t many on that list.) > > > You say that a streaming server can be installed on a virtual machine. > I don''t want to create any kind of disturb but I''ve asked the same > question to the KVM mailing list and they said that a streaming server > (I user red5), due to the high I/O traffic, it must absolutely not > virtualized. > >That may be true on KVM, I don''t know. Databases are usually the most brutal on VMs and I ran mysqlbench (all tests) on KVM as well as on the same machine native. KVM was within 5% of the native test. I did the same thing with Xen 3.4.0 and the performance was within 4% but when I moved my database storage to an LVM LV on DomU used as a hard drive in DomU the difference fell to less than 1%. Yes, the very IO dependent mysqlbench speed improved substatially when using LVM over disk files. The only real difference between mysqlbench on the real server and then on Xen/LVM was 20% higher CPU utilization. I will also add that if Xen has a bottleneck it''s traffic between DomUs. Don''t configure it in any way to have one DomU servicing another DomU unless both have real network cards attached to them. As strange as it sounds I''ve clocked 3x speed improvement by sending the data out of the machine to a switch and back in to another DomU as apposed to sending it straight across. There''s something messed up in that but if you follow this guideline you''ll get wire speed out of your DomUs. My advice, set up the streaming server and watch it. If it doesn''t work migrate it to new hardware. What are you out? The other note I''d add is that if your server is running at 100% utilization there''s no benefit in Virtualizing it. You will only know if it''s worth it by doing it and watching. Grant McWilliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Grant, I''m curious about your tests; LVM vs non LVM for the domU. Was your domU image based (sparse or no sparse) and using an LVM inside or did you set up your domU to have direct disk access? Was your dom0 LVM based as well? What non LVM fs were you using, ext2, ext3, XFS? - Brian On Sep 26, 2009, at 7:05 AM, Grant McWilliams wrote:> > On Sat, Sep 26, 2009 at 2:42 AM, Mauro <mrsanna1@gmail.com> wrote: > 2009/9/18 Jeff Sturm <jeff.sturm@eprize.com>: > > I don''t see why not. If you''re concerned about latency you could > always > > pin the server to a physical CPU. > > > > (Lately we''ve been doing a lot of physical-to-virtual migrations, > and > > I''ve found myself asking: "What kinds of servers do we have that > should > > never be virtual?" There aren''t many on that list.) > > > You say that a streaming server can be installed on a virtual machine. > I don''t want to create any kind of disturb but I''ve asked the same > question to the KVM mailing list and they said that a streaming server > (I user red5), due to the high I/O traffic, it must absolutely not > virtualized. > > > That may be true on KVM, I don''t know. Databases are usually the > most brutal on VMs and I ran mysqlbench (all tests) on > KVM as well as on the same machine native. KVM was within 5% of the > native test. I did the same thing > with Xen 3.4.0 and the performance was within 4% but when I moved my > database storage to an LVM > LV on DomU used as a hard drive in DomU the difference fell to less > than 1%. Yes, the very IO dependent > mysqlbench speed improved substatially when using LVM over disk > files. The only real difference between > mysqlbench on the real server and then on Xen/LVM was 20% higher CPU > utilization. > > I will also add that if Xen has a bottleneck it''s traffic between > DomUs. Don''t configure it in any way to have > one DomU servicing another DomU unless both have real network cards > attached to them. As strange as it > sounds I''ve clocked 3x speed improvement by sending the data out of > the machine to a switch and back in > to another DomU as apposed to sending it straight across. There''s > something messed up in that but if you follow > this guideline you''ll get wire speed out of your DomUs. > > My advice, set up the streaming server and watch it. If it doesn''t > work migrate it to new hardware. What are you out? > The other note I''d add is that if your server is running at 100% > utilization there''s no benefit in Virtualizing it. You will > only know if it''s worth it by doing it and watching. > > Grant McWilliams > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/9/26 Grant McWilliams <grantmasterflash@gmail.com>:> > I will also add that if Xen has a bottleneck it''s traffic between DomUs. > Don''t configure it in any way to have > one DomU servicing another DomU unless both have real network cards > attached to them. As strange as it > sounds I''ve clocked 3x speed improvement by sending the data out of the > machine to a switch and back in > to another DomU as apposed to sending it straight across. There''s something > messed up in that but if you follow > this guideline you''ll get wire speed out of your DomUs.Thank you for your answer it was very exhaustive. I''ve learned a new thing, that is the poor performance in traffic between domU in the same machine rather than domU in different machines. I''ve a domU with tomcat and many jruby applications and, in the same machine, another domU as database server with postgres. Based on what you say I''ve to reconfigure my systems to have domU in different machines. Perhaps I can leave domU in the same machine but assign a different NIC? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Sep-26 18:28 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sun, Sep 27, 2009 at 12:56 AM, Mauro <mrsanna1@gmail.com> wrote:> 2009/9/26 Grant McWilliams <grantmasterflash@gmail.com>: >> >> I will also add that if Xen has a bottleneck it''s traffic between DomUs. >> Don''t configure it in any way to have >> one DomU servicing another DomU unless both have real network cards >> attached to them.> Thank you for your answer it was very exhaustive. > I''ve learned a new thing, that is the poor performance in traffic > between domU in the same machine rather than domU in different > machines. > I''ve a domU with tomcat and many jruby applications and, in the same > machine, another domU as database server with postgres. > Based on what you say I''ve to reconfigure my systems to have domU in > different machines.Were you having network performance problems? There was a thread on this list some time ago about domU <-> domU network performance, and the ones that experience it were using newer pv_ops kernels while people who use 2.6.18 kernel (like RHEL/Centos5) or Suse -xen kernel got good performance. http://www.nabble.com/Fw%3A-Re%3A-bridge-throughput-problem-tt25328188.html#a25351569 Instead of having a second physical server just to get decent performance, I''d say it''s easier to simply change your dom0 kernel. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
peter_booth@mac.com
2009-Sep-26 18:35 UTC
Re: [Xen-users] streaming server on a virtual machine.
Mauro, Grant, Whenever innovative technology appears it seems inevitable that it is deployed in some contexts where it just doesn''t fit. I remember this happening with Corba, EJBs, ibject databases, web services, etc. If your priority is efficient use of machine resources (throughput) then Xen''s flexibility is amazing. If, on the other hand, you have a customer facing retail website then using Xen might reduce both hardware costs and market share as your customers move to faster competitor sites. Be certain that your physical architecture makes business sense! Peter Sent from my iPhone On Sep 26, 2009, at 1:56 PM, Mauro <mrsanna1@gmail.com> wrote:> 2009/9/26 Grant McWilliams <grantmasterflash@gmail.com>: >> >> I will also add that if Xen has a bottleneck it''s traffic between >> DomUs. >> Don''t configure it in any way to have >> one DomU servicing another DomU unless both have real network cards >> attached to them. As strange as it >> sounds I''ve clocked 3x speed improvement by sending the data out of >> the >> machine to a switch and back in >> to another DomU as apposed to sending it straight across. There''s >> something >> messed up in that but if you follow >> this guideline you''ll get wire speed out of your DomUs. > > Thank you for your answer it was very exhaustive. > I''ve learned a new thing, that is the poor performance in traffic > between domU in the same machine rather than domU in different > machines. > I''ve a domU with tomcat and many jruby applications and, in the same > machine, another domU as database server with postgres. > Based on what you say I''ve to reconfigure my systems to have domU in > different machines. > Perhaps I can leave domU in the same machine but assign a different > NIC? > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/9/26 Fajar A. Nugraha <fajar@fajar.net>:> On Sun, Sep 27, 2009 at 12:56 AM, Mauro <mrsanna1@gmail.com> wrote: >> 2009/9/26 Grant McWilliams <grantmasterflash@gmail.com>: >>> >>> I will also add that if Xen has a bottleneck it''s traffic between DomUs. >>> Don''t configure it in any way to have >>> one DomU servicing another DomU unless both have real network cards >>> attached to them. > >> Thank you for your answer it was very exhaustive. >> I''ve learned a new thing, that is the poor performance in traffic >> between domU in the same machine rather than domU in different >> machines. >> I''ve a domU with tomcat and many jruby applications and, in the same >> machine, another domU as database server with postgres. >> Based on what you say I''ve to reconfigure my systems to have domU in >> different machines. > > Were you having network performance problems? > There was a thread on this list some time ago about domU <-> domU > network performance, and the ones that experience it were using newer > pv_ops kernels while people who use 2.6.18 kernel (like RHEL/Centos5) > or Suse -xen kernel got good performance. > > http://www.nabble.com/Fw%3A-Re%3A-bridge-throughput-problem-tt25328188.html#a25351569 > > Instead of having a second physical server just to get decent > performance, I''d say it''s easier to simply change your dom0 kernel.I''m using debian lenny amd64 with kernel 2.6.26 with dom0 and in all my domU. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Sep-26 20:02 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sun, Sep 27, 2009 at 1:57 AM, Mauro <mrsanna1@gmail.com> wrote:> 2009/9/26 Fajar A. Nugraha <fajar@fajar.net>: >> On Sun, Sep 27, 2009 at 12:56 AM, Mauro <mrsanna1@gmail.com> wrote:>>> I''ve learned a new thing, that is the poor performance in traffic >>> between domU in the same machine rather than domU in different >>> machines.>> Were you having network performance problems? >> There was a thread on this list some time ago about domU <-> domU >> network performance, and the ones that experience it were using newer >> pv_ops kernels while people who use 2.6.18 kernel (like RHEL/Centos5) >> or Suse -xen kernel got good performance.> I''m using debian lenny amd64 with kernel 2.6.26 with dom0 and in all my domU.Using -xen kernel? If so, you shouldn''t have the problem. You can test it using iperf (or other similar tool). -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/9/26 Fajar A. Nugraha <fajar@fajar.net>:> On Sun, Sep 27, 2009 at 1:57 AM, Mauro <mrsanna1@gmail.com> wrote: >> 2009/9/26 Fajar A. Nugraha <fajar@fajar.net>: >>> On Sun, Sep 27, 2009 at 12:56 AM, Mauro <mrsanna1@gmail.com> wrote: > >>>> I''ve learned a new thing, that is the poor performance in traffic >>>> between domU in the same machine rather than domU in different >>>> machines. > >>> Were you having network performance problems? >>> There was a thread on this list some time ago about domU <-> domU >>> network performance, and the ones that experience it were using newer >>> pv_ops kernels while people who use 2.6.18 kernel (like RHEL/Centos5) >>> or Suse -xen kernel got good performance. > >> I''m using debian lenny amd64 with kernel 2.6.26 with dom0 and in all my domU. > > Using -xen kernel? If so, you shouldn''t have the problem. You can test > it using iperf (or other similar tool). >Yes I must use xen kernel otherwise I can''t work with xen. Now I test with iperf I''ll know. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-27 14:19 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sat, Sep 26, 2009 at 10:56 AM, Mauro <mrsanna1@gmail.com> wrote:> > Thank you for your answer it was very exhaustive. > I''ve learned a new thing, that is the poor performance in traffic > between domU in the same machine rather than domU in different > machines. > I''ve a domU with tomcat and many jruby applications and, in the same > machine, another domU as database server with postgres. > Based on what you say I''ve to reconfigure my systems to have domU in > different machines. > Perhaps I can leave domU in the same machine but assign a different NIC? >Assigning a different NIC to it resolves the problem. Sending traffic out and back into the same machine seems to be fine. If you look at the Xen architecture it all makes sense. There an amazing number of steps between each DomU that packets have to go through. There are fewer between the DomU and an external switch. I''ve not rerun the test since 3.2 but have no reason to believe things have changed. Grant McWiliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-27 14:25 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sat, Sep 26, 2009 at 11:28 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Sun, Sep 27, 2009 at 12:56 AM, Mauro <mrsanna1@gmail.com> wrote: > > 2009/9/26 Grant McWilliams <grantmasterflash@gmail.com>: > >> > >> I will also add that if Xen has a bottleneck it''s traffic between DomUs. > >> Don''t configure it in any way to have > >> one DomU servicing another DomU unless both have real network cards > >> attached to them. > > > Thank you for your answer it was very exhaustive. > > I''ve learned a new thing, that is the poor performance in traffic > > between domU in the same machine rather than domU in different > > machines. > > I''ve a domU with tomcat and many jruby applications and, in the same > > machine, another domU as database server with postgres. > > Based on what you say I''ve to reconfigure my systems to have domU in > > different machines. > > Were you having network performance problems? > There was a thread on this list some time ago about domU <-> domU > network performance, and the ones that experience it were using newer > pv_ops kernels while people who use 2.6.18 kernel (like RHEL/Centos5) > or Suse -xen kernel got good performance. > > > http://www.nabble.com/Fw%3A-Re%3A-bridge-throughput-problem-tt25328188.html#a25351569 > > Instead of having a second physical server just to get decent > performance, I''d say it''s easier to simply change your dom0 kernel. > > -- > Fajar >The domU to domU performance problem that I extensively tested and documented was with the 2.6.18 without pv_ops. The problem does not appear if the traffic is routed straight out of the box. I saw full GB speeds from a DomU to any other machine. I''d get about 350-400 Mb/s between DomUs. After changing every variable and digging through the xen-users and xen-devel archives the answer I got was yes, that does exist and the solution was to assign a NIC to the domU or only have DomUs communicate with DomUs on other physical machines. This was with Xen 3.2.1. I haven''t rerun my testing with 3.4 yet because I worked around it. Grant McWilliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: Mauro [mailto:mrsanna1@gmail.com] > Sent: Saturday, September 26, 2009 5:43 AM > To: Jeff Sturm > Cc: xen-users > Subject: Re: [Xen-users] streaming server on a virtual machine. > > > (Lately we''ve been doing a lot of physical-to-virtual migrations, and > > I''ve found myself asking: "What kinds of servers do we have that should > > never be virtual?" There aren''t many on that list.) > > > You say that a streaming server can be installed on a virtual machine. > I don''t want to create any kind of disturb but I''ve asked the same > question to the KVM mailing list and they said that a streaming server > (I user red5), due to the high I/O traffic, it must absolutely not > virtualized.Honestly I know little about streaming servers--I work with database servers each day. If I can guess though that they behave somewhat like a web server with large media files, it seems intuitive that: - Large files will be often read in large sequential chunks; random I/O will be limited - Disk bandwidth usage shouldn''t exceed network bandwidth usage (since the purpose of reading files from disk is to serve them over rtmp). Given the latter, assuming common 1Gbps Ethernet is used, it doesn''t seem likely that you''d need much more than 100MB/s read throughput from your disks during peak load. I know from experience this is well within the capabilities of a paravirtualized Linux guest (on Xen) when the dom0 has a good I/O subsystem. However, if someone who knows red5 far better than I can point on the flaws in my simplistic analysis, feel free to tell me I''m out to lunch... It may also be true this isn''t feasible on KVM. I know even less about KVM than streaming servers. On the surface it appears to be a virtualization product architecturally similar to Xen but with hypervisor integrated in kernel, and somewhat less mature than Xen. But if the disk block drivers on KVM are paravirtualized as they are in Xen, I don''t see why KVM could not reach the same I/O performance levels (if not today, then someday). I also want to point out that using absolutes like "...must absolutely not..." doesn''t convey any useful information to understand the tradeoffs. It may or may not be a stupid idea to virtualize a streaming server, but I''d bet it is certainly possible, if not downright feasible. Ultimately it comes down to cost vs. performance. The prices of commodity components on the open market can fluctuate from day to day, and this is sometimes a bigger factor in a buying decision than the capabilities of the technology. If you have the opportunity to evaluate some solutions, test and quote out a few different configurations. Examples: 1) a farm of physical servers with direct-attached storage 2) a farm of physical servers connected to an iSCSI SAN 3) a farm of virtual servers running on a few large systems with a FC SAN The first option may be the simplest to configure and operate, but may increase your overall storage costs depending on how much storage capacity you require and how many servers (bandwidth) are needed. Option 2 enables sharing files, e.g. with NFS or a cluster filesystem, which may reduce the overall storage capacity needed, and hence the expense. Option 3 has the priciest individual components, with larger servers and a FC SAN (typically much more expensive than iSCSI), but depending on how much you can reduce the number of physical nodes needed by sharing RAM/CPU/disk, could actually be cost-effective compared to the first two. What''s also interesting here is that 3) may yield higher peak I/O throughput to a single server (in spite of virtualization) than either 1) or 2), since 1) is limited to the performance of the local disk, and 2) limits per-host throughput to the iSCSI network interface or HBA. I can''t speak to your application, but it''s possible that peak performance is just as important to consider as overall aggregate throughput. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-27 14:42 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sat, Sep 26, 2009 at 9:48 AM, Brian Krusic <brian@krusic.com> wrote:> Hi Grant, > I''m curious about your tests; LVM vs non LVM for the domU. > > Was your domU image based (sparse or no sparse) and using an LVM inside or > did you set up your domU to have direct disk access? > > Was your dom0 LVM based as well? > > What non LVM fs were you using, ext2, ext3, XFS? > > - Brian > >The DomU for the disk-file test was a non-sparse image file set up as a drive with straight partitions inside like this ''tap:aio:/srv/xen/webserver.img,xvdb,w'',. The DomU for the LVM test had an LV in the Dom0 set up as a drive in the DomU like this ''phy:/dev/vgsys/lvweb,xvdb,w'' As you probably know that you''d never use a sparse image file for anything performance related (at least until the hard drive was full!). Both were using ext3. All the variables were the same, same OS drive for both (non-sparse file disk image) and the Dom0 was the same. I just replaced the disk file "drive" with the LVM "drive" at the same mountpoint and re-ran the tests. I had to convince SELinux in both cases that the new location was OK for mysql to write to so an SELinux policy was created that allowed this. I''d always had apache running out of an LV but mysql was writing to the Disk File. After I ran these tests I''m moved mysql over to the LV as well. I would say though that even though the performance between native and real is virtually nothing you still need to pay attention to the CPU utilization differences. Running the db in the VM is still consuming more CPU than if it were on a bare machine. It''s also worth noting that previously I was working on contract on a full suite testing of the various VT technologies and during those tests I documented disk speed tests between Disk files and LVM and to my surprise I found very little in the way of speed increases of LVM over Disk files (from parity to 10% max). This was not expected at all. I believet the speed increase with the LVM volume was due to fewer IOs needed. mysqlbench is very tough on IOs. Grant McWilliams Some people, when confronted with a problem, think "I know, I''ll use Windows." Now they have two problems. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
why would fewer i/o''s be needed when accessing lvm vs non lvm if both underlying fs was ext3? i''m not challenging, just asking as this could effect how i look at things, i would ofcourse verify this as well. you have the awsome advantage of testing intricate setups which is valuable. - Brian On Sep 27, 2009, at 7:42 AM, Grant McWilliams wrote:> On Sat, Sep 26, 2009 at 9:48 AM, Brian Krusic <brian@krusic.com> > wrote: > Hi Grant, > > I''m curious about your tests; LVM vs non LVM for the domU. > > Was your domU image based (sparse or no sparse) and using an LVM > inside or did you set up your domU to have direct disk access? > > Was your dom0 LVM based as well? > > What non LVM fs were you using, ext2, ext3, XFS? > > - Brian > > > The DomU for the disk-file test was a non-sparse image file set up > as a drive with straight partitions inside like this ''tap:aio:/srv/ > xen/webserver.img,xvdb,w'',. The DomU for the LVM test had an LV in > the Dom0 set up as a drive in the DomU like this ''phy:/dev/vgsys/ > lvweb,xvdb,w'' > > As you probably know that you''d never use a sparse image file for > anything performance related (at least until the hard drive was > full!). Both were using ext3. All the variables were the same, same > OS drive for both (non-sparse file disk image) and the Dom0 was the > same. I just replaced the disk file "drive" with the LVM "drive" at > the same mountpoint and re-ran the tests. I had to convince SELinux > in both cases that the new location was OK for mysql to write to so > an SELinux policy was created that allowed this. I''d always had > apache running out of an LV but mysql was writing to the Disk File. > After I ran these tests I''m moved mysql over to the LV as well. > > I would say though that even though the performance between native > and real is virtually nothing you still need to pay attention to the > CPU utilization differences. Running the db in the VM is still > consuming more CPU than if it were on a bare machine. > > It''s also worth noting that previously I was working on contract on > a full suite testing of the various VT technologies and during those > tests I documented disk speed tests between Disk files and LVM and > to my surprise I found very little in the way of speed increases of > LVM over Disk files (from parity to 10% max). This was not expected > at all. I believet the speed increase with the LVM volume was due to > fewer IOs needed. mysqlbench is very tough on IOs. > > Grant McWilliams > > Some people, when confronted with a problem, think "I know, I''ll use > Windows." > Now they have two problems. > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-27 23:55 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Sun, Sep 27, 2009 at 2:55 PM, Brian Krusic <brian@krusic.com> wrote:> why would fewer i/o''s be needed when accessing lvm vs non lvm if both > underlying fs was ext3? > i''m not challenging, just asking as this could effect how i look at things, > i would ofcourse verify this as well. > > you have the awsome advantage of testing intricate setups which is > valuable. > > - Brian > >Brian, I''ll tell you my theory and if anyone knows for sure can pipe up. If you are accessing an ext3 filesystem in a diskfile inside an ext3 filesystem on a disk the OS(s) will have more work to do than accessing an ext3 filesystem on a disk. With the LVM method you''re passing through the disk (via phy:) to the VM so it''s accessing it directly. I don''t really think it has anything to do with LVM, that''s just how I passed it to the VM. Grant McWilliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ah, this leads me to my next ? Do VMs need exclusive access to disk or can I have a common space on the dom0 with multiple domUs accessing it via phy mechanism? - Brian On Sep 27, 2009, at 4:55 PM, Grant McWilliams wrote:> On Sun, Sep 27, 2009 at 2:55 PM, Brian Krusic <brian@krusic.com> > wrote: > why would fewer i/o''s be needed when accessing lvm vs non lvm if > both underlying fs was ext3? > > i''m not challenging, just asking as this could effect how i look at > things, i would ofcourse verify this as well. > > you have the awsome advantage of testing intricate setups which is > valuable. > > - Brian > > > Brian, > > I''ll tell you my theory and if anyone knows for sure can pipe > up. If you are accessing an ext3 filesystem in a diskfile inside an > ext3 filesystem on a disk the OS(s) will have more work to do than > accessing an ext3 filesystem on a disk. With the LVM method you''re > passing through the disk (via phy:) to the VM so it''s accessing it > directly. I don''t really think it has anything to do with LVM, > that''s just how I passed it to the VM. > > Grant McWilliams_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Eisenberg
2009-Sep-28 09:10 UTC
RE: [Xen-users] streaming server on a virtual machine.
From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Brian Krusic Sent: Sunday, September 27, 2009 9:16 PM To: Grant McWilliams Cc: Jeff Sturm; xen-users; Mauro Subject: Re: [Xen-users] streaming server on a virtual machine. Ah, this leads me to my next ? Do VMs need exclusive access to disk or can I have a common space on the dom0 with multiple domUs accessing it via phy mechanism? - Brian Not via PHY - it''s a block device just like any other - multiple simultaneous accesses will corrupt the device unless you have something like DRBD in place. Probably easier is just an NFS mount. Best Regards, Nathan Eisenberg _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-28 09:19 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Mon, Sep 28, 2009 at 2:10 AM, Nathan Eisenberg <nathan@atlasnetworks.us>wrote:> ** > > Ah, this leads me to my next ? > > Do VMs need exclusive access to disk or can I have a common space on the > dom0 with multiple domUs accessing it via phy mechanism? > > - Brian > > > > Not via PHY – it’s a block device just like any other – multiple > simultaneous accesses will corrupt the device unless you have something like > DRBD in place. Probably easier is just an NFS mount. > > > > Best Regards, > > Nathan Eisenberg > > And even then be careful of more than one VM having write access to afile or you''ll end up with a mess. There are many conversations here on the Xen lists about how to do this. It''s not an uncommon thing to want multiple VMs booting and running off one source location. Grant McWilliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Brian Krusic wrote:> Ah, this leads me to my next ? > > Do VMs need exclusive access to disk or can I have a common space on the > dom0 with multiple domUs accessing it via phy mechanism?Hi Brian, Yes. Dom0 uses LVM and passes through LVs to the paravirts. All domUs share a single /usr partition that is mounted read only. ''phy:vg0/usr,hda2,r'' There has never been a problem that I''ve noticed and I have been doing this since 3.0.2-2. (you can tell by the "hda" disk name :) Mike Wright _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-28 17:12 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Mon, Sep 28, 2009 at 8:52 AM, Mike Wright <mike.wright@mailinator.com>wrote:> Brian Krusic wrote: > >> Ah, this leads me to my next ? >> >> Do VMs need exclusive access to disk or can I have a common space on the >> dom0 with multiple domUs accessing it via phy mechanism? >> > > Hi Brian, > > Yes. Dom0 uses LVM and passes through LVs to the paravirts. All domUs > share a single /usr partition that is mounted read only. > > ''phy:vg0/usr,hda2,r'' > > There has never been a problem that I''ve noticed and I have been doing this > since 3.0.2-2. (you can tell by the "hda" disk name :) > > Mike Wright > >The key words here are "read only". Do not allow any domUs to write to the same LV. Grant McWilliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, 2009-09-28 at 10:12 -0700, Grant McWilliams wrote:> The key words here are "read only". Do not allow any domUs to write > to the same LV.How do you facilitate providing updates to the domU''s, for example, when package upgrades need to take place? John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thats a great question. I provide my own repos which sync to Centos/Plus/RPMForge/Gitco upon my choosing, normally once a month. I then test any new updates before going live with a foreach loop, yumming hosts in my list. I did the same is an MS$ env and am unsure of a better way. But I''ve never managed more then 1,000 nodes. Any one care to share there methodology? - Brian On Sep 28, 2009, at 10:26 AM, John Madden wrote:> On Mon, 2009-09-28 at 10:12 -0700, Grant McWilliams wrote: >> The key words here are "read only". Do not allow any domUs to write >> to the same LV. > > How do you facilitate providing updates to the domU''s, for example, > when > package upgrades need to take place? > > John > > > > -- > John Madden > Sr UNIX Systems Engineer > Ivy Tech Community College of Indiana > jmadden@ivytech.edu > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2009-Sep-28 17:58 UTC
Re: [Xen-users] streaming server on a virtual machine.
On Mon, Sep 28, 2009 at 10:39 AM, Brian Krusic <brian@krusic.com> wrote:> Thats a great question. > > I provide my own repos which sync to Centos/Plus/RPMForge/Gitco upon my > choosing, normally once a month. > > I then test any new updates before going live with a foreach loop, yumming > hosts in my list. > > I did the same is an MS$ env and am unsure of a better way. But I''ve never > managed more then 1,000 nodes. > > Any one care to share there methodology? > > - Brian > > > On Sep 28, 2009, at 10:26 AM, John Madden wrote: > > On Mon, 2009-09-28 at 10:12 -0700, Grant McWilliams wrote: >> >>> The key words here are "read only". Do not allow any domUs to write >>> to the same LV. >>> >> >> How do you facilitate providing updates to the domU''s, for example, when >> package upgrades need to take place? >> >> John >> >>I think what he''s probably asking more than anything is since you have multiple OSs running off a single LV who gets the job of updating? I would assume that you have to shutdown the other domains (maybe pause?), remount /usr as read/write, do the updates using one DomU and the unpause the others. I think this would be problemic though because you just updated one portion of the OS and didn''t touch anything else (configs etc..). This whole method makes me a bit nervous. Either that or you just never update! Could you give us more detail on your update procedure please. Grant McWilliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Eisenberg
2009-Sep-28 19:42 UTC
RE: [Xen-users] streaming server on a virtual machine.
-----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Brian Krusic Sent: Monday, September 28, 2009 10:40 AM To: John Madden Cc: Grant McWilliams; xen-users; Mike Wright Subject: Re: [Xen-users] streaming server on a virtual machine. Thats a great question. I provide my own repos which sync to Centos/Plus/RPMForge/Gitco upon my choosing, normally once a month. I then test any new updates before going live with a foreach loop, yumming hosts in my list. I did the same is an MS$ env and am unsure of a better way. But I''ve never managed more then 1,000 nodes. Any one care to share there methodology? - Brian --------------------------------- This is precisely what I do, and it works very well. I have ''multithreaded'' my script so that several systems update in parallel, so that it doesn''t take -forever-. I''ve also broken it out into a couple processes for clustered environments (I have a lot of web clusters) so that only one web server in a cluster ever updates at once. In essence, my methodology for handling updates to domUs matches my general attitude about them: treat them like normal machines, and use the oldschool voodoo handed down from our fore-admins to manage them. This wheel has already been built. :-) Best Regards, Nathan Eisenberg _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Cool. How do you multithread a script? Is it just a matter of launching them in the back ground and going on to the next host w/o waiting for the script to complete? SAs wanna know :) - Brian On Sep 28, 2009, at 12:42 PM, Nathan Eisenberg wrote:> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com > ] On Behalf Of Brian Krusic > Sent: Monday, September 28, 2009 10:40 AM > To: John Madden > Cc: Grant McWilliams; xen-users; Mike Wright > Subject: Re: [Xen-users] streaming server on a virtual machine. > > Thats a great question. > > I provide my own repos which sync to Centos/Plus/RPMForge/Gitco upon > my choosing, normally once a month. > > I then test any new updates before going live with a foreach loop, > yumming hosts in my list. > > I did the same is an MS$ env and am unsure of a better way. But I''ve > never managed more then 1,000 nodes. > > Any one care to share there methodology? > > - Brian > --------------------------------- > > This is precisely what I do, and it works very well. I have > ''multithreaded'' my script so that several systems update in > parallel, so that it doesn''t take -forever-. I''ve also broken it > out into a couple processes for clustered environments (I have a lot > of web clusters) so that only one web server in a cluster ever > updates at once. > > In essence, my methodology for handling updates to domUs matches my > general attitude about them: treat them like normal machines, and > use the oldschool voodoo handed down from our fore-admins to manage > them. This wheel has already been built. :-) > > Best Regards, > Nathan Eisenberg > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Brian Krusic > Sent: Monday, September 28, 2009 3:52 PM > To: Nathan Eisenberg > Cc: xen-users > Subject: Re: [Xen-users] streaming server on a virtual machine. > > How do you multithread a script?We do it with tools like mssh: http://sourceforge.net/projects/mssh/ -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Eisenberg
2009-Sep-28 20:32 UTC
RE: [Xen-users] streaming server on a virtual machine.
-----Original Message----- From: Jeff Sturm [mailto:jeff.sturm@eprize.com] Sent: Monday, September 28, 2009 1:12 PM To: Brian Krusic; Nathan Eisenberg Cc: xen-users Subject: RE: [Xen-users] streaming server on a virtual machine.> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Brian Krusic > Sent: Monday, September 28, 2009 3:52 PM > To: Nathan Eisenberg > Cc: xen-users > Subject: Re: [Xen-users] streaming server on a virtual machine. > > How do you multithread a script?We do it with tools like mssh: http://sourceforge.net/projects/mssh/ -Jeff ----------------------- Oooh, I like that! I might have to switch to that. I put ''s around multithreading because it''s a little... shall we say... creative the way I did it. But MSSH looks cool! Thanks Jeff! Best Regards, Nathan Eisenberg _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users