Xen users - Dec 2007 - RAID-1 strategy for a Xen/CentOS server?

Hi,

I''m planning to migrate my current server configuration:

    AMD Athlon/X2
    FreeBSD 62-Release
    2 x 250GB HW RAID1 (all partitions RAIDed, incl. swap)

to a Xen/CentOS config:

    AMD Athlon/X2
    Dom0
        CentOS 51
        Xen 31
    Dom1/1
        FreeBSD 7
    Dom1/2
        CentOS 51
    ...

To "best preserve" RAID1 mirroring protections of OS, Data, etc

    *AND*

optimize performance, how should I setup RAID for the "new"
configurations?

Should DOM0 be RAIDed?

Each DOM1, as well?

Should I continue to use HW RAID or switch to SW?  A mix?


I understand I can do any/all ... choosing the "right" option is the
goal,
here.

Advice is appreciated.

Regards,

Bob Tompkins


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>
> To "best preserve" RAID1 mirroring protections of OS, Data, etc
>
>    *AND*
>
> optimize performance, how should I setup RAID for the "new"
configurations?
because vbd are slower than "real-world" devices, I believe you want
to
move the mirroring into dom0, so that the domU''s aren''t
sending the same
data through the hypervisor twice... beyond that, all the rules are the 
same as they would be in a non-XEN config. (This just makes plain sense, 
there''s less work to be done.)

There may be situations where it makes more sense to do the raid in the 
domUs, but you listed "optimize performance".

-Tom

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Tom,


because vbd are slower than "real-world" devices, I believe you want
to
> move the mirroring into dom0, so that the domU''s aren''t
sending the same
> data through the hypervisor twice... beyond that, all the rules are the
> same as they would be in a non-XEN config. (This just makes plain sense,
> there''s less work to be done.)
>
> There may be situations where it makes more sense to do the raid in the
> domUs, but you listed "optimize performance".

Thanks for the reply.

Yes, performance is my primary concern, at least during my planning.  (One
DomU
 will consist of Zimbra/CentOS, which I understand to be an I/O hog, so
I''m
trying to remove what bottlenecks I can> It still remains to be seen, but
Zimbra *itself* may be on eo fthose bottleneck that gets removed.)

As for moving the mirroring into DomU, generally, I''ve been slowly
coming to
the same conclusion.

Am I correct in my understanding that each DomU contains an instance of
OS/kernel + app binaries in it''s own virtual volume/file space, but
that
*data* (effectively, *any* dynamic content) is written by the DomU processes
to/from the Dom0 hypervisor''s volume/file space?

How about swap?  Dom0 of course has its swap -- and could/should be RAIDed,
but what about  GuestOS'' swap?  I honestly haven''t gotten that
far yet ....

And, if DomU hosts the RAID mirror, what''s the recommended file system
choice -- or is that dicated by Xen as a preference (I haven''t got to
that
either yet ...)

Regards,

Bob Tompkins


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 7 Dec 2007, Bob Tomkins wrote:
> Hi Tom,
>
>
> because vbd are slower than "real-world" devices, I believe you
want to
>> move the mirroring into dom0, so that the domU''s
aren''t sending the same
>> data through the hypervisor twice... beyond that, all the rules are the
>> same as they would be in a non-XEN config. (This just makes plain
sense,
>> there''s less work to be done.)
>>
>> There may be situations where it makes more sense to do the raid in the
>> domUs, but you listed "optimize performance".
>
>
> Thanks for the reply.
>
> Yes, performance is my primary concern, at least during my planning.  (One
> DomU
> will consist of Zimbra/CentOS, which I understand to be an I/O hog, so
I''m
I was going to say "add memory", and that''s always a help
with respect to
reducing disk I/O ... but zimbra seems to be a mail-server platform ... 
and MTA software often _forces_ disk I/Os via fsync() type calls. Still 
making sure that the VM has lots of memory will reduce the disk reads, and 
allow non sync()''d I/O to be cached for a bit longer and written to
disk
when convenient.

frankly, IMHO and in my experience, things that eat up disk I/O aren''t 
particularily well suited to running on virtual machines. XEN generally(*) 
is for cutting up a big machine into smaller pieces which can be more 
conveniently administered... partly due to the isolation between 
machines...

... but disk drives do not "isolate" well unless you are dedicating 
spindles to individual domains... if a domain starts maxing out it''s 
attached drives, any other domain using that drive is going to see it''s
load average go up as processes trying to use those drives get stuck in a 
long queue waiting for disk I/O.

(* - there are situations where inserting a "shim" between the
physical
hardware and the O/S are usefull... being able to backup a windows box, or 
replicate it via DRBD in real time are examples, that require having a 
"virtual" layer. The consistency of the virtual machine is also good,
as
you can pick up a domU and start it on another physical box and see the 
same virtual machine.)
> Am I correct in my understanding that each DomU contains an instance of
> OS/kernel + app binaries in it''s own virtual volume/file space,
but that
> *data* (effectively, *any* dynamic content) is written by the DomU
processes
> to/from the Dom0 hypervisor''s volume/file space?
>
> How about swap?  Dom0 of course has its swap -- and could/should be RAIDed,
> but what about  GuestOS'' swap?  I honestly haven''t gotten
that far yet ....
What is the point in "raid''ing" swap space? There''s
nothing in swap you
really need to preserve. If your target is trying to increase 
reliability... I''d find other approaches ... but even so, the same
logic
applies, either have the VM swap to a dedicated vbd or have it do LVM on 
it''s existing vbd''s and swap to one of those ... the same raid
rules
apply.

This might be relevent to your HW vs SW raid. Generally HW raid will make 
it simpler to replace a drive without having to reboot the system... but 
it tends to have constraints of it''s own...
> And, if DomU hosts the RAID mirror, what''s the recommended file
system
> choice -- or is that dicated by Xen as a preference (I haven''t got
to that
> either yet ...)
AFAIK, none of this has anything to do with xen. In theory the folks 
behind your target software (like zimbra) should have recommendations.

-Tom


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Tom,

On Dec 7, 2007 12:05 PM, Tom Brown <xensource.com@vmail.baremetal.com>
wrote:
> I was going to say "add memory", and that''s always a
help with respect to
> reducing disk I/O ... but zimbra seems to be a mail-server platform ...
> and MTA software often _forces_ disk I/Os via fsync() type calls. Still
> making sure that the VM has lots of memory will reduce the disk reads, and
> allow non sync()''d I/O to be cached for a bit longer and written
to disk
> when convenient.
>
You''re correct re: Zimbra''s disk I/O intensity --- whther or
not  that''s an
obstacle that "more memory" will overcome, or mitigate, I simply do
not know
at this point.

FWIW, "this box" has 4GB RAM atm, expandale to 8GB.
>
> frankly, IMHO and in my experience, things that eat up disk I/O
aren''t
> particularily well suited to running on virtual machines. XEN generally(*)
> is for cutting up a big machine into smaller pieces which can be more
> conveniently administered... partly due to the isolation between
> machines...
>
That''s my goal.  In particular, isolating Zimbra which I understand has
some
"quirks & oddities" that I''d rather not have
"polluting" the rest of my
Production server

> ... but disk drives do not "isolate" well unless you are
dedicating
> spindles to individual domains... if a domain starts maxing out
it''s
> attached drives, any other domain using that drive is going to see
it''s
> load average go up as processes trying to use those drives get stuck in a
> long queue waiting for disk I/O.
>
Clear.

(* - there are situations where inserting a "shim" between the
physical
> hardware and the O/S are usefull... being able to backup a windows box, or
> replicate it via DRBD in real time are examples, that require having a
> "virtual" layer. The consistency of the virtual machine is also
good, as
> you can pick up a domU and start it on another physical box and see the
> same virtual machine.)

Clear.

What is the point in "raid''ing" swap space? There''s
nothing in swap you
> really need to preserve. If your target is trying to increase
> reliability...

It is.  And the answer to your question seems to vary greatly depending on
to whom you listen
.  In my case, the reference is:

http://www.linuxjournal.com/article/5898

"If you are using RAID-1 to help to ensure that your system stays up in the
event of a hard disk partition failure, you should consider raiding your
swap partition(s). If the disk or partition you are using for swap goes bad,
your machine may crash. Using a RAID-1 device for a swap partition can help
prevent that crash. If one of the mirrored swap partitions goes bad, the
kernel automatically will fail over to the other, and your system should
keep running until you can fix the disk problem."

> I''d find other approaches ... but even so, the same logic
> applies, either have the VM swap to a dedicated vbd or have it do LVM on
> it''s existing vbd''s and swap to one of those ... the same
raid rules
> apply.
>
> This might be relevent to your HW vs SW raid. Generally HW raid will make
> it simpler to replace a drive without having to reboot the system... but
> it tends to have constraints of it''s own...

I gather the Devil''s in the Details of the prior two paragraphs.  In
principle I understand, but have yet to touch-and-feel in practice.

My broad-brushstroke goals -- for the whole system,as well as for individual
DomUs -- include:

(1) Maximize failover capability
(2) Minimize potential fordata loss due to HW failure
(3) Maximize the performance of the system

(I''ve given up on running at less than 10W total power utilization ...
;-) )

> > And, if DomU hosts the RAID mirror, what''s the recommended
file system
> > choice -- or is that dicated by Xen as a preference (I
haven''t got to
> that
> > either yet ...)
>
> AFAIK, none of this has anything to do with xen. In theory the folks
> behind your target software (like zimbra) should have recommendations.

You''re correct -- in answering my misstated question.  I *meant* to
ask:
What''s the recommended FS for the Dom0 Xen host?

Regards,

Bob


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>
> FWIW, "this box" has 4GB RAM atm, expandale to 8GB.
IMHO that means nothing... if it''s handling 4 mailboxes, it''s
overkill, if
it''s handling 40,000 you''re probably dead before you even
started... but
then you could come up with particular examples where the roles of that 4 
and 40,000 could be reversed... e.g. support@microsoft.com, 
abuse@hotmail.com, etc versus 40,000 never used freebie accounts that are 
included with some other product...
> "If you are using RAID-1 to help to ensure that your system stays up
in the
> event of a hard disk partition failure, you should consider raiding your
> swap partition(s).
No argument. Generally my thinking with respect to raid 1 is to reduce the 
chance of data loss, but as per the introductory clause, there can 
be different goals.
> My broad-brushstroke goals -- for the whole system,as well as for
individual
> DomUs -- include:
>
> (1) Maximize failover capability
??? By definition, "failover" requires redundant hardware, sort of the
opposite of cutting up a big machine into smaller pieces (xen). If you 
want failover capability, see linux-ha.org ... and you need to be able to 
get at the data from the redundant hardware... it''s a rather different 
picture than a normal server setup.

Stuffing DRBD under the vbd''s so that a redundant server can mirror the
block devices is fine start :) ... but that certainly isn''t what I
thought
you meant by hardware vs software RAID... and you''re entering the world
of
network attached storage....
> (2) Minimize potential fordata loss due to HW failure
> (3) Maximize the performance of the system
>
> (I''ve given up on running at less than 10W total power utilization
... ;-) )
>
>
> You''re correct -- in answering my misstated question.  I *meant*
to ask:
> What''s the recommended FS for the Dom0 Xen host?
it is basically irrelevent, as anything that will boot xen is pretty much 
sufficient. dom0 isn''t supposed to do much work (beyond having the
kernel
relay I/O requests to the real hardware). If you''re using file images 
instead of block devices for your domU vbd''s then it may be relevent...
but ext3 should be fine. I can''t really imagine a scenario where the
more
esoteric filesystems would be needed... but I''m no expert, and when you
get into journalling filesystems (in the domUs) there are some complex 
interactions (e.g. preserving the order of writes, which is one reason why 
using real block devices instead of files may be better).

(Then again, using disk files for filesystem images probably conflicts 
with maximizing failover capabilities... this may depend on how many domUs 
you have, and whether you need/want to be able to fail them over 
independently.)

-Tom

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

??? By definition, "failover" requires redundant hardware,


Yes.  And I since I''m considering RAID1, I have 2, redundant hardDrives
on
this box.

Maximizing failover to the extent possible on this box addresses *DISK*
failover, and nothing more.

you meant by hardware vs software RAID...


Here, simply the differences -- in application & utility -- between the SATA
RAID-1 provided  "on the motherboard" (hardware), versus the options
of
Software RAID Mirroring (e.g., http://aplawrence.com/Linux/softmirror.html,
FreeBSD''s Vimum/GEOM, etc etc)

> it is basically irrelevent

LOL.  Clearly you''ve not been a contributor to some of the fanatical
this-or-that-filesystem arguments I''ve stumbeld across.

but ext3 should be fine.


It sounds like there''s no specific requirement.  Hence, agreed.

> I can''t really imagine a scenario where the more
> esoteric filesystems would be needed... but I''m no expert, and
when you
> get into journalling filesystems (in the domUs) there are some complex
> interactions (e.g. preserving the order of writes, which is one reason why
> using real block devices instead of files may be better).
>
> (Then again, using disk files for filesystem images probably conflicts
> with maximizing failover capabilities... this may depend on how many domUs
> you have, and whether you need/want to be able to fail them over
> independently.)

I''m at a point where I need to (re)read what I''ve found and
consider well
your comments/question.  That''s what weekends are good for!

Regards,

Bob


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Am I correct in my understanding that each DomU contains an instance of
> OS/kernel + app binaries in it''s own virtual volume/file space,
but that
> *data* (effectively, *any* dynamic content) is written by the DomU
> processes to/from the Dom0 hypervisor''s volume/file space?
Basically, yes.

The domU writes data to a virtual device which is provided, in some way, by 
dom0.  If you''re using HVM then your guest might be using an emulated
version
of a PATA drive (for instance).  If you''re using PV then the guest is
using
an optimised, Xen-aware block driver to access the virtual device.  PV block 
devices have better performance and can also be used in some HVM guests; for 
non-Linux HVM you''re basically stuck with the slower emulator.

You can basically export anything that''s a block device in dom0 as a
virtual
disk, including a RAIDed volume, LVM volumes, etc.

As Tom said, doming RAID in dom0 and hiding it from the guest will halve the 
number of disk requests that the domUs have to issue.  It''ll also
simplify
the configuration of the domUs (if you change the RAID setup in dom0 they 
don''t have to know about it).
> How about swap?  Dom0 of course has its swap -- and could/should be RAIDed,
> but what about  GuestOS'' swap?  I honestly haven''t gotten
that far yet ....
If you''re using RAID to keep the host up whilst a disk drive explodes,
then
RAIDing the guest swap too may make sense.  Guests do all their own swapping, 
so if they have swap configured then it is a necessary part of their running 
state.
> And, if DomU hosts the RAID mirror, what''s the recommended file
system
> choice -- or is that dicated by Xen as a preference (I haven''t got
to that
> either yet ...)
Pretty much any filesystem is good...  There have been some problems with XFS 
and Xen so possibly I wouldn''t recommend that.  ext3 seems to be widely
held
as the most solid Linux filesystem these days, if not the most technically 
advanced, so I guess if you don''t have any specific reason to choose
another
one then that would be my recommendation.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users