Xen users - May 2009 - Looking for Xen success stories in a production data center environment

Hi All,

I am looking for some data regarding successful Xen deployment in production
data center environments.  What I need are hard facts regarding platform
stability and I/O performance metrics.  I''ve found some comparisons
(Vmware
vs Xen) on the web but a lot of them seem a bit dated.

Just to save bandwidth please don''t post subjective "works great
for us"
type messages - this isn''t a poll :-)

What I, and probably others, need is whitepaper level objective uptime and
performance metric info.

Many, many thanks.

Joe
-- 
View this message in context:
http://www.nabble.com/Looking-for-Xen-success-stories-in-a-production-data-center-environment-tp23454153p23454153.html
Sent from the Xen - User mailing list archive at Nabble.com.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, May 08, 2009 at 02:40:45PM -0700, Joe Armstrong
wrote:
> 
> Hi All,
> 
> I am looking for some data regarding successful Xen deployment in
production
> data center environments.  What I need are hard facts regarding platform
> stability and I/O performance metrics.  I''ve found some
comparisons (Vmware
> vs Xen) on the web but a lot of them seem a bit dated.
> 
> Just to save bandwidth please don''t post subjective "works
great for us"
> type messages - this isn''t a poll :-)
> 
> What I, and probably others, need is whitepaper level objective uptime and
> performance metric info.
> 
> Many, many thanks.
> 
Hopefully someone can come up with benchmarks/numbers.

You could always run your own benchmarks? :-)

-- Pasi

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen wrote:
> 
> 
> Hopefully someone can come up with benchmarks/numbers.
> 
> You could always run your own benchmarks? :-)
> 
> 
Yes - I did that first thing... and got number for  both disk and network
I/O to within 1 or 2 % of native... nice.

But, my little setup is not even close to a "real data center" setup,
and no
way I can get stability data in a few days of running small tests.  I was
hoping somebody (Oh, I don''t know, maybe from Amazon...) would cough up
some
really good data on stability & I/O performance.  Or maybe somebody out
there would know where to find it.  I''m looking but not a lot of luck
so
far.  But if I find some I''ll post it.

Thanks.

Joe
-- 
View this message in context:
http://www.nabble.com/Looking-for-Xen-success-stories-in-a-production-data-center-environment-tp23454153p23485301.html
Sent from the Xen - User mailing list archive at Nabble.com.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

QqpI have real production data, some ppsitive, some negative. What is  
your context?

Are you talking about web app or batch?
Are you most concerned about throughput or respense time?
Are you concerned with self-managed or VPS/cloud environments?

The near-native observations you made don''t tell the whole story. My  
experience is that most production applications are built and deployed  
in unoptimized states such that over-resourcing is the norm. A cost  
that''s unique to virtualization is the erratic latencies due to  
hypervisor scheduling algorithm. In one example a rails app with  
consistent response times of 400ms native showed response times of  
400msec to 3sec virtualized, depending on %steal values. There are a  
couple of recent papers reporting similar data.

Sent from my iPhone
App
On May 11, 2009, at 11:11 AM, Joe Armstrong <Joe.Armstrong@webex.com>  
wrote:
>
>
>
> Pasi Kärkkäinen wrote:
>>
>>
>> Hopefully someone can come up with benchmarks/numbers.
>>
>> You could always run your own benchmarks? :-)
>>
>>
>
> Yes - I did that first thing... and got number for  both disk and  
> network
> I/O to within 1 or 2 % of native... nice.
>
> But, my little setup is not even close to a "real data center"  
> setup, and no
> way I can get stability data in a few days of running small tests.   
> I was
> hoping somebody (Oh, I don''t know, maybe from Amazon...) would
cough
> up some
> really good data on stability & I/O performance.  Or maybe somebody  
> out
> there would know where to find it.  I''m looking but not a lot of  
> luck so
> far.  But if I find some I''ll post it.
>
> Thanks.
>
> Joe
> -- 
> View this message in context:
http://www.nabble.com/Looking-for-Xen-success-stories-in-a-production-data-center-environment-tp23454153p23485301.html
> Sent from the Xen - User mailing list archive at Nabble.com.
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

-----Original Message-----
From: Peter Booth [mailto:peter_booth@mac.com] 
Sent: Monday, May 11, 2009 8:48 AM
To: Joe.Armstrong@webex.com
Cc: xen-users@lists.xensource.com
Subject: Re: [Xen-users] Looking for Xen success stories in a production data
center environment

QqpI have real production data, some ppsitive, some negative. What is  
your context?

Are you talking about web app or batch?
Are you most concerned about throughput or respense time?
Are you concerned with self-managed or VPS/cloud environments?

The near-native observations you made don''t tell the whole story. My  
experience is that most production applications are built and deployed  
in unoptimized states such that over-resourcing is the norm. A cost  
that''s unique to virtualization is the erratic latencies due to  
hypervisor scheduling algorithm. In one example a rails app with  
consistent response times of 400ms native showed response times of  
400msec to 3sec virtualized, depending on %steal values. There are a  
couple of recent papers reporting similar data.

Sent from my iPhone
App
----

Hi Peter,

We are looking to run a linux-based mail server together with its web-based
front-end, windows-based Active Directory server, and several other windows and
linux-based products associated with the mail server.  The mail server will have
lots of small file-based and network-based I/O and occasionally medium sized I/O
depending on attachments.  Not dealing with streaming data or real-time
responses.   This will be a self-managed environment (via a professional
operations staff).

I have heard some hints of troubles if the number of VM''s exceeds the
number of cores - but internally we think we can manage that problem by not
overloading a given physical platform.

I understand that we may have to give up some amount of performance for the
better manageability & recoverability of a virtualized environment -
that''s OK.  Performance-wise I think we can tweak when needed and even
run some components on physical hardware if needed.  Any insight you might have
in that area would be greatly appreciated.

As for stability, I have no idea... I have heard Xen is stable, but at this
point I have no hard evidence of that - either for or against.  And I
can''t measure stability via some benchmark - that only comes from
long-term running of a service.  I just don''t know where to get that
data.  Any clue on that ?

Thanks.

Joe

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Joe Armstrong wrote:
>As for stability, I have no idea... I have heard Xen is stable, but 
>at this point I have no hard evidence of that - either for or 
>against.
I''ve been running a few small servers on Xen for a while now - about 
6 months. I''ve had a couple of unexplained guest failures*, but to be 
honest no more (probably less) than we had when they were all 
separate boxes. None of the servers is heavily loaded - being a DNS 
master, DNS resolver, low traffic MailMan list server, and a fairly 
low traffic ''LAMP'' webserver - so I can''t honestly
give any
indications as to performance (and all the separate boxes Xen 
replaced were old ''hand me downs'' with varied specs).

Even the Xen host machines are hand me downs that I''ve got because 
they don''t have the hardware support needed for Microsoft''s 
virtualisation stuff.


* Guest is consuming 100% CPU but is completely unresponsive to 
network or xen console.

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, May 13, 2009 at 2:46 AM, Simon Hobson <linux@thehobsons.co.uk>
wrote:
> Joe Armstrong wrote:
>
>> As for stability, I have no idea... I have heard Xen is stable, but at
>> this point I have no hard evidence of that - either for or against.
>
> I''ve been running a few small servers on Xen for a while now -
about 6
> months. I''ve had a couple of unexplained guest failures*, but to
be honest
> no more (probably less) than we had when they were all separate boxes.
I''m using mostly Linux PV domU for production purposes. Most guest
failures was due to not enough RAM assigned to domU which triggered
OOM killer, so it''s not really Xen-specific problem.

I have one Xen domU which is up for 280 days, RHEL4 x86_64, using
custom xenified 2.6.16 kernel, from an old version of Xen (3.1,  I
think). Last downtime for that domU was to extend its disk. This host
is mostly lightly-loaded, serving local yum http mirror, with high
load around midnight (doing reposync, repomanage, repotrack,
createrepo, etc.).

Another long-running domU is RHEL5 x86_64, kernel
2.6.18-92.1.13.el5xen, running Lotus Domino 8.0.1, up for 189 days.
Last downtime was for migrating to this physical host to get more
resource (cpu, disk, etc.) and upgrading domU kernel.

Recently I started using HVM Windows domU with GPLPV. The longest one
was up since Jan/31 2009, no problems so far.
Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 11, 2009 at 08:11:52AM -0700, Joe Armstrong
wrote:
> 
> 
> 
> Pasi Kärkkäinen wrote:
> > 
> > 
> > Hopefully someone can come up with benchmarks/numbers.
> > 
> > You could always run your own benchmarks? :-)
> > 
> > 
> 
> Yes - I did that first thing... and got number for  both disk and network
> I/O to within 1 or 2 % of native... nice.
> 
> But, my little setup is not even close to a "real data center"
setup, and no
> way I can get stability data in a few days of running small tests.  I was
> hoping somebody (Oh, I don''t know, maybe from Amazon...) would
cough up some
> really good data on stability & I/O performance.  Or maybe somebody out
> there would know where to find it.  I''m looking but not a lot of
luck so
> far.  But if I find some I''ll post it.
> 
$ xm info
system                 : Linux
host                   : hostname
release                : 2.6.11-xen0
version                : #1 Mon Apr 4 13:51:34 EEST 2005
machine                : i686
cores                  : 2
hyperthreads_per_core  : 1
cpu_mhz                : 996
memory                 : 1663
free_memory            : 1259

$ uptime
 09:01:18 up 880 days, 16:48,  1 user,  load average: 0.00, 0.00, 0.00

That''s old Xen 2.0 host running a couple of domUs.. no problems at all
with stability :)
last time it was down because of fan getting broken and the heat crashing the
host..


Another Xen 3.0 host:

# xm info
host                   : hostname.localdomain
release                : 2.6.18-92.1.10.el5xen
version                : #1 SMP Tue Aug 5 08:46:32 EDT 2008
machine                : i686
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 3010
hw_caps                :
bfebfbff:20000000:00000000:00000180:0000e43d:00000000:00000001
total_memory           : 4095
free_memory            : 83
node_to_cpu            : node0:0-1
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-92.1.10.el5
xen_caps               : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Aug  5 07:34:25 EDT 2008
xend_config_format     : 2

# uptime
 09:05:07 up 271 days, 20:48,  1 user,  load average: 0.01, 0.01, 0.00

That one is running multiple Linux domUs.. both debian and rhel/centos.
271 days ago it was rebooted because of Xen (centos) upgrade.

Hopefully others can give you more numbers/comments.

-- Pasi

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users