Hi Everyone,
I''m looking for a solution to run some Xen VM''s with fairly
significant(up
to ~30TB) local storage volumes. We plan to use two Sun "Thumper"
x4540''s
with 48*1TB drives (8 controllers).
I have a few questions, and would appreciate feedback before I try to
achieve something dumb or impossible :)
- Has Xen been tested / does Xen support very large volumes? We will
probably end up using ext3 (as it seems its the only supported option), so
no single volume in a VM will exceed ~16TB, but we will be at this end of
the spectrum on several volumes.
- Can anyone suggest a better option than creating 7 RAID5 arrays of 6
disks each (1 per controller); using the remaining 6 drives for RAID-10 for
DomU and hot spares and then adding the 7 RAID5 volumes to a LVM group,
creating a LV per Xen VM block device? Has anybody used mdadm and/or LVM to
handle this number of drives?
- We are required to be able to bring up VMs on one machine on another
within a reasonable period of time (this is defined as days, not hours) and
to keep the backup copy reasonably in sync (again, days not hours). Can
anyone comment on the suitability of rsync to copy such massive block
devices? We would use version 3, which I know has improved performance for
this sort of move, but is it going to literally take forever to checksum
48TB? If anyone is able to give example of rsync performance (same switch or
crossover - whichever we conclude is fastest) over gigabit Ethernet, that
would be a bonus. We expect there to be significant spare resources
available to domU at night for checksumming if required.
- I know XFS is not supported by RHEL, but does it work - and/or well?
I''ve seen several reports that suggest it performs far better (best
summary
http://hepix.caspur.it/storage/hep_pdf/2007/Spring/Petkus_HEPiX_Spring06.storageeval.pdf)
but what is it like for reliability?
- Does anyone has any suggestions / improvements to this plan? Has
anybody done this before - with similar hardware?
- [An alternative scenario, if we can get it, would be a third Thumper -
in which case we would run half the VMs on each of two Thumpers running
RAID-10 (8 RAID-1 volumes, each a PV) for ~24TB per machine, and use the
third Thumper, partitioned as above with RAID-5 for more like RAID-5 levels
of storage efficiency, for backups of VMs and disaster recovery. Comments on
this would be fantastic too.]
The VMs are expected to be quite IO-heavy, hence the requirement for local
storage (not iSCSI) and para virtualization.
Any comments much appreciated.
Many thanks,
Alex
Hi Everyone,
I''m looking for a solution to run some Xen VM''s with fairly
significant(up
to ~30TB) local storage volumes. We plan to use two Sun "Thumper"
x4540''s
with 48*1TB drives (8 controllers), and Red Hat Enterprise Linux 5 (x86_64).
I have a few questions, and would appreciate feedback before I try to
achieve something dumb or impossible :)
- Has Xen been tested / does Xen support very large volumes? We will
probably end up using ext3 (as it seems its the only supported option), so
no single volume in a VM will exceed ~16TB, but we will be at this end of
the spectrum on several volumes.
- Can anyone suggest a better option than creating 7 RAID5 arrays of 6
disks each (1 per controller); using the remaining 6 drives for RAID-10 for
DomU and hot spares and then adding the 7 RAID5 volumes to a LVM group,
creating a LV per Xen VM block device? Has anybody used mdadm and/or LVM to
handle this number of drives?
- We are required to be able to bring up VMs on one machine on another
within a reasonable period of time (this is defined as days, not hours) and
to keep the backup copy reasonably in sync (again, days not hours). Can
anyone comment on the suitability of rsync to copy such massive block
devices? We would use version 3, which I know has improved performance for
this sort of move, but is it going to literally take forever to checksum
48TB? If anyone is able to give example of rsync performance (same switch or
crossover - whichever we conclude is fastest) over gigabit Ethernet, that
would be a bonus. We expect there to be significant spare resources
available to domU at night for checksumming if required.
- I know XFS is not supported by RHEL, but does it work - and/or well?
I''ve seen several reports that suggest it performs far better (best
summary
http://hepix.caspur.it/storage/hep_pdf/2007/Spring/Petkus_HEPiX_Spring06.storageeval.pdf)
but what is it like for reliability?
- Does anyone has any suggestions / improvements to this plan? Has
anybody done this before - with similar hardware?
- [An alternative scenario, if we can get it, would be a third Thumper -
in which case we would run half the VMs on each of two Thumpers running
RAID-10 (8 RAID-1 volumes, each a PV) for ~24TB per machine, and use the
third Thumper, partitioned as above with RAID-5 for more like RAID-5 levels
of storage efficiency, for backups of VMs and disaster recovery. Comments on
this would be fantastic too.]
The VMs are expected to be quite IO-heavy, hence the requirement for local
storage (not iSCSI) and para virtualization.
Any comments much appreciated.
Many thanks,
Alex
[this email previously posted to fedora-xen@redhat.com; posted here after a
suggestion there :(]
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Sat, Sep 06, 2008 at 11:35:13AM +0200, Alex Davies wrote:> > I have a few questions, and would appreciate feedback before I try to > achieve something dumb or impossible :) > > - I know XFS is not supported by RHEL, but does it work - and/or well? > I''ve seen several reports that suggest it performs far better (best summary > http://hepix.caspur.it/storage/hep_pdf/2007/Spring/Petkus_HEPiX_Spring06.storageeval.pdf) > but what is it like for reliability?I think XFS has issues on 32 bit linux kernels, related to 4k stacks.. and people have been running it on 64b linux kernels with success. I''m not running XFS myself, so this is just what I''ve read from the internets:) -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> * We are required to be able to bring up VMs on one machine on > another within a reasonable period of time (this is defined as > days, not hours) and to keep the backup copy reasonably in > sync (again, days not hours). Can anyone comment on the > suitability of rsync to copy such massive block devices? We > would use version 3, which I know has improved performance for > this sort of move, but is it going to literally take forever > to checksum 48TB? If anyone is able to give example of rsync > performance (same switch or crossover - whichever we conclude > is fastest) over gigabit Ethernet, that would be a bonus. We > expect there to be significant spare resources available to > domU at night for checksumming if required.Use block replication software to replicate block devices. gnbd, for example, will work fine.> The VMs are expected to be quite IO-heavy, hence the requirement for > local storage (not iSCSI) and para virtualization.Then take another look at your storage setup -- depending on what you mean by "IO-heavy," those boxes may not perform all that well. Are you talking thousands of small i/o''s, which would be bad, or hundreds of large i/o''s, which would probably be fine? John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sep 5, 2008, at 13:37, Alex Davies wrote:> using the remaining 6 drives for RAID-10 for DomUUnless you meant "for Dom0", then beware that software RAID-10, LVM and Xen doesn''t work: https://bugzilla.redhat.com/show_bug.cgi?id=223947 https://bugzilla.redhat.com/show_bug.cgi?id=224077 -- http://develooper.com/ - http://askask.com/
Thank-you everyone for your suggestions; some extremely useful points have been raised for which I am very grateful. To answer Erin''s questions: we are running RHEL/Xen because we have a business requirement for a decent number of RHEL virtual machines (~30). We could run ESX (or KVM, or...) but we (I) am also keen to try to use Xen for real - i''ve used ESX a lot. There are also licensing benefits for my company that make Xen on RHEL > ESX / KVM. I would love to use ZFS, but unfortunately my Solaris foo is ~0, and consequently we shall have to stick to Linux/ext3. RedHat assure me that if I keep each filesystem to <10T this is supported and encouraged. Based on feedback from this post (and elsewhere), we have decided to go for 2 ISCSI boxes and 2 Thumpers. Two of our virtual machines require >30T of storage, and no redundancy, so we will run one VM on each Thumper (with local storage). The other ~30 VMs will run using the ISCSI volumes to allow live migration. I''m sure i''ll be posting here when it comes to getting this lot working, but if anyone has any advice for this setup please let me know! Many thanks, Alex On Thu, Sep 11, 2008 at 6:55 AM, Ask Bjørn Hansen <ask@develooper.com> wrote:> > On Sep 5, 2008, at 13:37, Alex Davies wrote: > >> using the remaining 6 drives for RAID-10 for DomU > > Unless you meant "for Dom0", then beware that software RAID-10, LVM and Xen > doesn''t work: > > https://bugzilla.redhat.com/show_bug.cgi?id=223947 > https://bugzilla.redhat.com/show_bug.cgi?id=224077 > > -- > http://develooper.com/ - http://askask.com/ > > >-- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently.
On Sep 14, 2008, at 9:16, Alex Davies wrote:> Two of our virtual machines require >30T of storage, and no > redundancy, so we will run one VM on each Thumper (with local > storage).Are you replacing the 500GB disks with 1TB disks in the Thumbers? Unless "no redundancy" really truly means "it''s fine if it breaks a lot", then please keep in mind the numbers. 40+ disks are that many times more likely to fail than one disk. Don''t just stripe all the disks into one big volume. Either keep them as separate devices so they fail individually or use RAID5 or (better) RAID6 and keep plenty spares ready. Your plan of doing RAID in sets of 8 or so disks is probably about right. - ask -- http://develooper.com/ - http://askask.com/