Hi I''m thinking about using Xen over vmware in small production environment. Basically one vm will be used for a web app and the other will be used for the db. The db will have one spindle/disc dedicated to it for improved I/O over file based storage. Traffic is somewhat light, probably 75,000 request a day. Is anybody running Xen in a production environment? If so what is maintenance like? Are you constantly having to keep a careful watch? Does it crash often? Run out of memory? OR is has it been a very blissful experience and your always confident that it''s up and running. I''m just trying to get a broad overview of your experiences with Xen. Many thanks for your time and knowledge, Matt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Matthew Palmer
2007-Jun-30 00:04 UTC
[Xen-users] Re: Is Anybody Running Xen in Production Environment
On Fri, Jun 29, 2007 at 08:14:44AM -0700, Matias wrote:> Is anybody running Xen in a production environment?Yep, responsible for serving about 80 million requests per day.> If so what is maintenance like? Are you constantly > having to keep a careful watch? Does it crash > often? Run out of memory? OR is has it been > a very blissful experience and your always confident > that it''s up and running.I''ve been quite impressed -- it''s never really been a hassle. Nagios keeps an eye on things, and we''re running heartbeat on redundant machines to make sure that even if something goes pop we''re still covered. Even with all that, I can''t think of a production failure (even non-customer-impacting) that has definitely been Xen''s fault. - Matt -- Software engineering: that part of computer science which is too difficult for the computer scientist. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nico Kadel-Garcia
2007-Jun-30 00:18 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
Matthew Palmer wrote:> On Fri, Jun 29, 2007 at 08:14:44AM -0700, Matias wrote: > >> Is anybody running Xen in a production environment? >> > > Yep, responsible for serving about 80 million requests per day >Me, too. I''ve been using it for development systems, QA environments, and some production services. The instabilities have been due to ongoing software development, now that the "install a modified glibc" requirements seem to have vanished with the tweaks of the latest kernels under RHEL and CentOS.>> If so what is maintenance like? Are you constantl >> having to keep a careful watch? Does it crash >> often? Run out of memory? OR is has it been >> a very blissful experience and your always confident >> that it''s up and running. >> > > I''ve been quite impressed -- it''s never really been a hassle. Nagios keeps > an eye on things, and we''re running heartbeat on redundant machines to make > sure that even if something goes pop we''re still covered. Even with all > that, I can''t think of a production failure (even non-customer-impacting) > that has definitely been Xen''s fault. >What are you using to configure your Nagios? I''ve been working with fruity, that seems to be a nice tool but need completion. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Matthew Palmer
2007-Jun-30 00:59 UTC
[Xen-users] Re: Is Anybody Running Xen in Production Environment
[Please don''t Cc me on posts to mailing lists -- I read the list] On Sat, Jun 30, 2007 at 01:18:58AM +0100, Nico Kadel-Garcia wrote:> Matthew Palmer wrote: > >On Fri, Jun 29, 2007 at 08:14:44AM -0700, Matias wrote: > >>If so what is maintenance like? Are you constantl > >>having to keep a careful watch? Does it crash > >>often? Run out of memory? OR is has it been > >>a very blissful experience and your always confident > >>that it''s up and running. > > > >I''ve been quite impressed -- it''s never really been a hassle. Nagios keeps > >an eye on things, and we''re running heartbeat on redundant machines to make > >sure that even if something goes pop we''re still covered. Even with all > >that, I can''t think of a production failure (even non-customer-impacting) > >that has definitely been Xen''s fault. > > What are you using to configure your Nagios?A text editor. If you use hostgroups extensively as we do, it''s really simple. We''ve got maybe 50 object definitions for our 600 or so service checks. - Matt -- How about "suspender snapping three martini lunching mahogany tabled conference room equipped with overhead projector dwelling golden parachute flying bill gates specifying buzzword spewing computerworld and datamation reading trend bandwagoneering meeting going morons". -- Tom O''Toole _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ian Marlier
2007-Jun-30 03:13 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
On 6/29/07 8:59 PM, "Matthew Palmer" <mpalmer@hezmatt.org> wrote:> [Please don''t Cc me on posts to mailing lists -- I read the list] > > On Sat, Jun 30, 2007 at 01:18:58AM +0100, Nico Kadel-Garcia wrote: >> Matthew Palmer wrote: >>> On Fri, Jun 29, 2007 at 08:14:44AM -0700, Matias wrote: >>>> If so what is maintenance like? Are you constantl >>>> having to keep a careful watch? Does it crash >>>> often? Run out of memory? OR is has it been >>>> a very blissful experience and your always confident >>>> that it''s up and running. >>> >>> I''ve been quite impressed -- it''s never really been a hassle. Nagios keeps >>> an eye on things, and we''re running heartbeat on redundant machines to make >>> sure that even if something goes pop we''re still covered. Even with all >>> that, I can''t think of a production failure (even non-customer-impacting) >>> that has definitely been Xen''s fault. >> >> What are you using to configure your Nagios? > > A text editor. If you use hostgroups extensively as we do, it''s really > simple. We''ve got maybe 50 object definitions for our 600 or so service > checks.Same here. I''ve got production systems from the extreme management backend to the customer-facing side of things running on xen machines, and they''ve been totally stable. Heartbeat adds an extra level of security for the really mission-critical stuff. Managing the nagios text config files has always been less of pain than managing a config tool, for me. 70+ hosts can be boiled down to 10 or so host groups, and then services.cfg isn''t too complicated at all. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mike Peters
2007-Jun-30 12:22 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
On Sat, 30 Jun 2007 01:18:58 +0100 Nico Kadel-Garcia <nkadel@gmail.com> wrote:> Matthew Palmer wrote: > > On Fri, Jun 29, 2007 at 08:14:44AM -0700, Matias wrote: > > > >> Is anybody running Xen in a production environment? > >> > > > > Yep, responsible for serving about 80 million requests per day > > > Me, too. I''ve been using it for development systems, QA environments, > and some production services. The instabilities have been due to > ongoing software development, now that the "install a modified glibc" > requirements seem to have vanished with the tweaks of the latest > kernels under RHEL and CentOS.Just adding to the list of happy customers. I run about 70-80 SUSE domains on openSUSE 10.2, similarly hosting various development, QA testing, demonstration and both critical and non-critical production servers. Only problems we''ve had so far have been some hardware issues totally non-Xen related and some problems with our own software, again not Xen related.> > >> If so what is maintenance like? Are you constantl > >> having to keep a careful watch? Does it crash > >> often? Run out of memory? OR is has it been > >> a very blissful experience and your always confident > >> that it''s up and running. > >> > > > > I''ve been quite impressed -- it''s never really been a hassle. > > Nagios keeps an eye on things, and we''re running heartbeat on > > redundant machines to make sure that even if something goes pop > > we''re still covered. Even with all that, I can''t think of a > > production failure (even non-customer-impacting) that has > > definitely been Xen''s fault. > What are you using to configure your Nagios? I''ve been working with > fruity, that seems to be a nice tool but need completion. >I have my own scripts for building and manipulating domains. These also update the Nagios config files as necessary. I use a good old text editor for anything beyond that. If you get the initial set up well organised it really isn''t too difficult to maintain. -- Mike Web Site: http://www.ice2o.com JabberID: mpeters@jabber.org Registered Linux User #247123 <n3tg0d> has /usr/bin/emacs been put into /etc/shells yet? :P _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mike Peters
2007-Jun-30 12:27 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
On Sat, 30 Jun 2007 01:18:58 +0100 Nico Kadel-Garcia <nkadel@gmail.com> wrote:> Matthew Palmer wrote: > > On Fri, Jun 29, 2007 at 08:14:44AM -0700, Matias wrote: > > > >> Is anybody running Xen in a production environment? > >> > > > > Yep, responsible for serving about 80 million requests per day > > > Me, too. I''ve been using it for development systems, QA environments, > and some production services. The instabilities have been due to > ongoing software development, now that the "install a modified glibc" > requirements seem to have vanished with the tweaks of the latest > kernels under RHEL and CentOS.Just adding to the list of happy customers. I run about 70-80 SUSE domains on openSUSE 10.2, similarly hosting various development, QA testing, demonstration and both critical and non-critical production servers. Only problems we''ve had so far have been some hardware issues totally non-Xen related and some problems with our own software, again not Xen related.> > >> If so what is maintenance like? Are you constantl > >> having to keep a careful watch? Does it crash > >> often? Run out of memory? OR is has it been > >> a very blissful experience and your always confident > >> that it''s up and running. > >> > > > > I''ve been quite impressed -- it''s never really been a hassle. > > Nagios keeps an eye on things, and we''re running heartbeat on > > redundant machines to make sure that even if something goes pop > > we''re still covered. Even with all that, I can''t think of a > > production failure (even non-customer-impacting) that has > > definitely been Xen''s fault. > What are you using to configure your Nagios? I''ve been working with > fruity, that seems to be a nice tool but need completion. >I have my own scripts for building and manipulating domains. These also update the Nagios config files as necessary. I use a good old text editor for anything beyond that. If you get the initial set up well organised it really isn''t too difficult to maintain. -- Mike Web Site: http://www.ice2o.com JabberID: mpeters@jabber.org Registered Linux User #247123 <n3tg0d> has /usr/bin/emacs been put into /etc/shells yet? :P _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Harper
2007-Jun-30 12:39 UTC
RE: [Xen-users] Re: Is Anybody Running Xen in Production Environment
> > Just adding to the list of happy customers. I run about 70-80 > SUSE domains on openSUSE 10.2, similarly hosting various development,QA> testing, demonstration and both critical and non-critical production > servers. Only problems we''ve had so far have been some hardware issues > totally non-Xen related and some problems with our own software, again > not Xen related. >Just out of interest, did running Xen make the diagnosis of these issues any harder? Particularly the hardware side of things? I''ve just set up Xen 3.1.0 on a HP DL385, running Debian Etch in Dom0, a spam filter (Debian Etch), a secondary nameserver (Debian Etch), a second windows domain controller (Windows 2K3R2E), a small business server (Windows SBS2K3R2S), and a web server (Windows 2K3R2E). Currently Dom0 isn''t running any of HP''s (closed source binary afaik) monitoring stuff, and I''m not sure if there is going to be any value in trying to make it go... I guess my concern is that if a disk, fan, psu, or anything else fails or something, I don''t yet have the tools to know about it (server will be in a colo facility). Are binary-only modules supported in Dom0? I suspect that there might be some problems... Thanks James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mike Peters
2007-Jun-30 12:55 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
On Sat, 30 Jun 2007 22:39:52 +1000 "James Harper" <james.harper@bendigoit.com.au> wrote:> > > > Just adding to the list of happy customers. I run about 70-80 > > SUSE domains on openSUSE 10.2, similarly hosting various > > development, > QA > > testing, demonstration and both critical and non-critical production > > servers. Only problems we''ve had so far have been some hardware > > issues totally non-Xen related and some problems with our own > > software, again not Xen related. > > > > Just out of interest, did running Xen make the diagnosis of these > issues any harder? Particularly the hardware side of things? > > I''ve just set up Xen 3.1.0 on a HP DL385, running Debian Etch in > Dom0, a spam filter (Debian Etch), a secondary nameserver (Debian > Etch), a second windows domain controller (Windows 2K3R2E), a small > business server (Windows SBS2K3R2S), and a web server (Windows > 2K3R2E). Currently Dom0 isn''t running any of HP''s (closed source > binary afaik) monitoring stuff, and I''m not sure if there is going to > be any value in trying to make it go... > > I guess my concern is that if a disk, fan, psu, or anything else fails > or something, I don''t yet have the tools to know about it (server will > be in a colo facility). Are binary-only modules supported in Dom0? I > suspect that there might be some problems... >In our particular case Xen has made no difference in this respect. We''re running this on Dell servers and all of the tools available to us in our non-Xen Linux environments are available on our Xen servers too. Unfortunately I don''t know about the HP stuff you mention. -- Mike Web Site: http://www.ice2o.com JabberID: mpeters@jabber.org Registered Linux User #247123 Telling the truth to people who misunderstand you is generally promoting a falsehood, isn''t it? -- A. Hope _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nuno Fernandes
2007-Jun-30 15:14 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
On Saturday 30 June 2007 13:39:52 James Harper wrote:> > Just adding to the list of happy customers. I run about 70-80 > > SUSE domains on openSUSE 10.2, similarly hosting various development, > > QA > > > testing, demonstration and both critical and non-critical production > > servers. Only problems we''ve had so far have been some hardware issues > > totally non-Xen related and some problems with our own software, again > > not Xen related. > > Just out of interest, did running Xen make the diagnosis of these issues > any harder? Particularly the hardware side of things? > > I''ve just set up Xen 3.1.0 on a HP DL385, running Debian Etch in Dom0, a > spam filter (Debian Etch), a secondary nameserver (Debian Etch), a > second windows domain controller (Windows 2K3R2E), a small business > server (Windows SBS2K3R2S), and a web server (Windows 2K3R2E). Currently > Dom0 isn''t running any of HP''s (closed source binary afaik) monitoring > stuff, and I''m not sure if there is going to be any value in trying to > make it go... > > I guess my concern is that if a disk, fan, psu, or anything else fails > or something, I don''t yet have the tools to know about it (server will > be in a colo facility). Are binary-only modules supported in Dom0? I > suspect that there might be some problems...Nops.. no modules from HP so far.. "RH EL 5 = Red Hat Enterprise Linux 5 HP will issue a customer advisory for RH EL 5 Xen. HP has certified and will support these servers listed below with the following caveats: * No HP management agent support * Greater than 64 GB system memory requires an updated kernel. Click here for more information * Guest OS support is provided by Red Hat. For more information on supported guest OS''s, see: www.redhat.com " from: http://h18007.www1.hp.com/support/files/server/us/download/26854.html?jumpid=reg_R1002_USEN We have 14 bl465c running a cluster of xen servers and so far so good :) Still wainting on hp drivers though... ./npf _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christian Horn
2007-Jul-01 09:12 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
On Sat, Jun 30, 2007 at 04:14:53PM +0100, Nuno Fernandes wrote:> Nops.. no modules from HP so far.. > > "RH EL 5 = Red Hat Enterprise Linux 5 > HP will issue a customer advisory for RH EL 5 Xen. HP has certified and will > support these servers listed below with the following caveats: > > * No HP management agent supportWonder who uses those anyway, with proprietary modules around this is one of the first things to point to when the kernel behaves strange. Getting aware of disk-failure in HP-servers was good enough for us and only needs tools operating in userspace. Did someone try to run array-info in a dom0 running on HP-hardware? Christian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nico Kadel-Garcia
2007-Jul-01 09:25 UTC
Re: [Xen-users] Re: Is Anybody Running Xen in Production Environment
Christian Horn wrote:> On Sat, Jun 30, 2007 at 04:14:53PM +0100, Nuno Fernandes wrote: > >> Nops.. no modules from HP so far.. >> >> "RH EL 5 = Red Hat Enterprise Linux 5 >> HP will issue a customer advisory for RH EL 5 Xen. HP has certified and will >> support these servers listed below with the following caveats: >> >> * No HP management agent support >> > Wonder who uses those anyway, with proprietary modules around this is > one of the first things to point to when the kernel behaves strange. > > Getting aware of disk-failure in HP-servers was good enough for us > and only needs tools operating in userspace. > Did someone try to run array-info in a dom0 running on HP-hardware? > >You''ve noticed that? I''m going through this now as part of negotiating some co-location services. They couldn''t *imagine* that I''d rather have backup OS images scattered around multiple servers than burning half of my available disk using RAID1, and couldn''t imagine using SNMP monitoring of overall services rather than these familiar integrated tools. It''s leading to a lot of... negotiation. It''s handy when you only have a very few staff, and a 99.9% uptime SLA, to get the reports of hardware outages rather than poking the OS''s. But given that development and QA work is so likely to slap down servers until a year or two after the software is stabilized and in use, it just seems a waste of time. Orient your support around fast swap-in capability, easy redeployment, and regular backups instead. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users