Chris ''Xenon'' Hanson
2009-May-29 20:30 UTC
[Xen-users] Advice on redundant SAN/NAS storage for Xen
This isn''t super Xen-specific, but it''s the basis on which I''d like to build my Xen cluster storage. I''m planning to expand my Xen servers at my datacenter into a cluster with high availability and reliability. As part of this, I want to move all DomU storage to a common SAN or NAS infrastructure and make all the Dom0s basically identical. In this way, I can move DomU''s around between Dom0s as needed for performance or reliability reasons. If a Dom0 server fails, I can just bring up its DomUs on different servers with no loss. So, to do this, I need fast block-access to a networked storage device. To avoid single-point of failure, I''d like this storage device to be a pair of discrete devices, each connected to all Dom0 machines via a pair of networks. This way, I can''t lose the whole thing through the failure of one storage device or network cables or switches. What I can''t put together in my mind is all the pieces. We have RAID and SAN and NAS and LVM and Xen and monkeys and bears oh my. The best design I can think of is this: Two machines running Linux configured as SANs, using something like ATA over Ethernet (AoE) to link them to a pair of GigE switches that then link to every Dom) box. The pair of SAN boxes each export a block of raw storage that the Dom0 machine then RAIDs together as RAID1 and provides to Xen and the DomU as a block device. The Dom0 gets network-portable storage, with RAID reliability and redundancy. The other way might be to have the Dom0 and Xen pass through both block devices to the DomU and let the DomU RAID them together. I''m not sure if either is better. Maybe RAID on the DomU would allow the DomU to be migrated easier? Is there a better and less messy way to provide redundant SAN-type storage to Xen DomUs? The main criteria are: Immune to failure of a single switch or SAN box. Allow DomUs to be moved seamlessly to other Dom0s without messy reconfiguration. Suggestions from anyone who has done it before? -- Chris ''Xenon'' Hanson, omo sanza lettere Xenon AlphaPixel.com PixelSense Landsat processing now available! http://www.alphapixel.com/demos/ "There is no Truth. There is only Perception. To Perceive is to Exist." - Xen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-May-30 05:30 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
On Sat, May 30, 2009 at 3:30 AM, Chris ''Xenon'' Hanson <xenon@alphapixel.com> wrote:> I''m planning to expand my Xen servers at my datacenter into a cluster with high > availability and reliability. As part of this, I want to move all DomU storage to a common > SAN or NAS infrastructure and make all the Dom0s basically identical. In this way, I can > move DomU''s around between Dom0s as needed for performance or reliability reasons. If a > Dom0 server fails, I can just bring up its DomUs on different servers with no loss.Simple goal, not-so-simple implementation.> The best design I can think of is this: > > Two machines running Linux configured as SANs, using something like ATA over Ethernet > (AoE) to link them to a pair of GigE switches that then link to every Dom) box. The pair > of SAN boxes each export a block of raw storage that the Dom0 machine then RAIDs together > as RAID1 and provides to Xen and the DomU as a block device. The Dom0 gets > network-portable storage, with RAID reliability and redundancy. > > The other way might be to have the Dom0 and Xen pass through both block devices to the > DomU and let the DomU RAID them together. I''m not sure if either is better. Maybe RAID on > the DomU would allow the DomU to be migrated easier?RAID might be the weakest link here. Think what will happen if : - one of the SAN box gets disconnected -> RAID will (hopefully) cope with it well and use the live SAN - some time later, the dead SAN is available again -> RAID won''t automatically re-add it - the other SAN dies. These are big IFs, but you get the idea.> > Is there a better and less messy way to provide redundant SAN-type storage to Xen DomUs? > The main criteria are: > > Immune to failure of a single switch or SAN box. > Allow DomUs to be moved seamlessly to other Dom0s without messy reconfiguration.Immune to a SAN box failure is hard. The common way to do it in enterprise-level storage is to have high availability in the SAN box. It does raid and have multiple controllers in a cluster/HA setup so that it''d be "immune" enough to disk or controller failure. I don''t think there''s a viable way to achieve that with your planned setup. Feel free to correct me if I wrong. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Andrew Lyon
2009-May-30 12:05 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
On Sat, May 30, 2009 at 6:30 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Sat, May 30, 2009 at 3:30 AM, Chris ''Xenon'' Hanson > <xenon@alphapixel.com> wrote: >> I''m planning to expand my Xen servers at my datacenter into a cluster with high >> availability and reliability. As part of this, I want to move all DomU storage to a common >> SAN or NAS infrastructure and make all the Dom0s basically identical. In this way, I can >> move DomU''s around between Dom0s as needed for performance or reliability reasons. If a >> Dom0 server fails, I can just bring up its DomUs on different servers with no loss. > > Simple goal, not-so-simple implementation. > >> The best design I can think of is this: >> >> Two machines running Linux configured as SANs, using something like ATA over Ethernet >> (AoE) to link them to a pair of GigE switches that then link to every Dom) box. The pair >> of SAN boxes each export a block of raw storage that the Dom0 machine then RAIDs together >> as RAID1 and provides to Xen and the DomU as a block device. The Dom0 gets >> network-portable storage, with RAID reliability and redundancy. >> >> The other way might be to have the Dom0 and Xen pass through both block devices to the >> DomU and let the DomU RAID them together. I''m not sure if either is better. Maybe RAID on >> the DomU would allow the DomU to be migrated easier? > > RAID might be the weakest link here. Think what will happen if : > - one of the SAN box gets disconnected -> RAID will (hopefully) cope > with it well and use the live SAN > - some time later, the dead SAN is available again -> RAID won''t > automatically re-add it > - the other SAN dies. > > These are big IFs, but you get the idea. > >> >> Is there a better and less messy way to provide redundant SAN-type storage to Xen DomUs? >> The main criteria are: >> >> Immune to failure of a single switch or SAN box. >> Allow DomUs to be moved seamlessly to other Dom0s without messy reconfiguration. > > Immune to a SAN box failure is hard. > The common way to do it in enterprise-level storage is to have high > availability in the SAN box. It does raid and have multiple > controllers in a cluster/HA setup so that it''d be "immune" enough to > disk or controller failure. I don''t think there''s a viable way to > achieve that with your planned setup. Feel free to correct me if I > wrong. > > -- > Fajar > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >Have a look at drdb, I''ve not used it myself but the idea of having two sets of disks (local or san) backing a single block device seems more robust than having two dom0''s accessing the same storage. http://www.gridvm.org/drbd-lvm-gnbd-and-xen-for-free-and-reliable-san.html http://lists.xensource.com/archives/html/xen-users/2008-11/msg00828.html http://openqrm.com/storage-cluster.png Andy _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Eisenberg
2009-May-30 20:19 UTC
RE: [Xen-users] Advice on redundant SAN/NAS storage for Xen
In my experience, effectively using DRBD is a major undertaking, and it can VERY easily cause you a lot of heartache. Approach with caution! Best Regards Nathan Eisenberg Sr. Systems Administrator Atlas Networks, LLC support@atlasnetworks.us http://support.atlasnetworks.us/portal -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Andrew Lyon Sent: Saturday, May 30, 2009 5:05 AM To: Fajar A. Nugraha; Chris ''Xenon'' Hanson Cc: Xen User-List Subject: Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen On Sat, May 30, 2009 at 6:30 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Sat, May 30, 2009 at 3:30 AM, Chris ''Xenon'' Hanson > <xenon@alphapixel.com> wrote: >> I''m planning to expand my Xen servers at my datacenter into a cluster with high >> availability and reliability. As part of this, I want to move all DomU storage to a common >> SAN or NAS infrastructure and make all the Dom0s basically identical. In this way, I can >> move DomU''s around between Dom0s as needed for performance or reliability reasons. If a >> Dom0 server fails, I can just bring up its DomUs on different servers with no loss. > > Simple goal, not-so-simple implementation. > >> The best design I can think of is this: >> >> Two machines running Linux configured as SANs, using something like ATA over Ethernet >> (AoE) to link them to a pair of GigE switches that then link to every Dom) box. The pair >> of SAN boxes each export a block of raw storage that the Dom0 machine then RAIDs together >> as RAID1 and provides to Xen and the DomU as a block device. The Dom0 gets >> network-portable storage, with RAID reliability and redundancy. >> >> The other way might be to have the Dom0 and Xen pass through both block devices to the >> DomU and let the DomU RAID them together. I''m not sure if either is better. Maybe RAID on >> the DomU would allow the DomU to be migrated easier? > > RAID might be the weakest link here. Think what will happen if : > - one of the SAN box gets disconnected -> RAID will (hopefully) cope > with it well and use the live SAN > - some time later, the dead SAN is available again -> RAID won''t > automatically re-add it > - the other SAN dies. > > These are big IFs, but you get the idea. > >> >> Is there a better and less messy way to provide redundant SAN-type storage to Xen DomUs? >> The main criteria are: >> >> Immune to failure of a single switch or SAN box. >> Allow DomUs to be moved seamlessly to other Dom0s without messy reconfiguration. > > Immune to a SAN box failure is hard. > The common way to do it in enterprise-level storage is to have high > availability in the SAN box. It does raid and have multiple > controllers in a cluster/HA setup so that it''d be "immune" enough to > disk or controller failure. I don''t think there''s a viable way to > achieve that with your planned setup. Feel free to correct me if I > wrong. > > -- > Fajar > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >Have a look at drdb, I''ve not used it myself but the idea of having two sets of disks (local or san) backing a single block device seems more robust than having two dom0''s accessing the same storage. http://www.gridvm.org/drbd-lvm-gnbd-and-xen-for-free-and-reliable-san.html http://lists.xensource.com/archives/html/xen-users/2008-11/msg00828.html http://openqrm.com/storage-cluster.png Andy _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Itamar Reis Peixoto
2009-May-30 22:15 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
freenas with iscsi ? On Sat, May 30, 2009 at 5:19 PM, Nathan Eisenberg <nathan@atlasnetworks.us> wrote:> In my experience, effectively using DRBD is a major undertaking, and it can VERY easily cause you a lot of heartache. Approach with caution! > > Best Regards > Nathan Eisenberg > Sr. Systems Administrator > Atlas Networks, LLC > support@atlasnetworks.us > http://support.atlasnetworks.us/portal >-- ------------ Itamar Reis Peixoto e-mail/msn: itamar@ispbrasil.com.br sip: itamar@ispbrasil.com.br skype: itamarjp icq: 81053601 +55 11 4063 5033 +55 34 3221 8599 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rudi Ahlers
2009-May-31 05:49 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
On Sat, May 30, 2009 at 10:19 PM, Nathan Eisenberg <nathan@atlasnetworks.us> wrote:> In my experience, effectively using DRBD is a major undertaking, and it can VERY easily cause you a lot of heartache. Approach with caution! > > Best Regards > Nathan Eisenberg > Sr. Systems Administrator > Atlas Networks, LLC > support@atlasnetworks.us > http://support.atlasnetworks.us/portalHi Nathan, Can you please elaborate on this? And my question, on this same topic, would be, what is wrong with using 2 standard servers (i.e. not a dedicated, expensive NAS device) with 2 or 4 HDD''s each, setup with something like DRBD / RedHat Cluster Suite to offer one single redundant NAS / SAN server? Or, even using something like http://www.openfiler.com/ or http://www.freenas.org/ ? -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-May-31 06:26 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
On Sun, May 31, 2009 at 12:49 PM, Rudi Ahlers <rudiahlers@gmail.com> wrote:> And my question, on this same topic, would be, what is wrong with > using 2 standard servers (i.e. not a dedicated, expensive NAS device) > with 2 or 4 HDD''s each, setup with something like DRBD / RedHat > Cluster Suite to offer one single redundant NAS / SAN server?Nothing''s wrong, if you understand how it works, plus its consequences. For example, there''s fencing issues on RHCS, which has caused frustration for some people. Plus it requires that you have a shared storage, which (from the example you give) you don''t. Can you emulate the shared storage using DRBD? Sure. But it introduces another layer of complexity. Plus you need synchronous primary-primary mode which introduced additional complexity, performance penalty, and (taken from drbd site) "These systems are very sensitive to failures of the replication network. Currently we cannot generally recommend this for production use" If you understand the requirements and its consequences, and decided that you can accept it, then it''s probably good for you. IMHO it''s not worthed. If I had that hardware I simply use them directly as dom0, with scheduled replication among each node. It does not provide automatic failover, nor support for live migration, but it greatly simplifies system setup and gives maximum I/O troughput.> Or, even > using something like http://www.openfiler.com/ or > http://www.freenas.org/ ?Their greatest feature is that they can convert an ordinary x86 server into iscsi SAN/NAS. They can''t use the example setup you give and turn it into an enterprise SAN replacement. If your main goal is domU migration, and you can live with the SAN/NAS being the single point of failure, then those are good choices. Personally I''d use opensolaris for their excellent (and supported) zvol feature. If you have enough budget you might also want to look at Sun''s Unified Storage. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Stratton
2009-May-31 13:49 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
On Sun, 31 May 2009, Fajar A. Nugraha wrote:> Can you emulate the shared storage using DRBD? Sure. But it introduces > another layer of complexity. Plus you need synchronous > primary-primary mode which introduced additional complexity, > performance penalty, and (taken from drbd site) "These systems are > very sensitive to failures of the replication network. Currently we > cannot generally recommend this for production use"Some of that is a bit dated, we have been running DRBD in 4 production 10 TB clusters for over a year now.><>Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-May-31 14:17 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
On Sun, May 31, 2009 at 8:49 PM, Nathan Stratton <nathan@robotics.net> wrote:> Some of that is a bit dated, we have been running DRBD in 4 production 10 TB > clusters for over a year now.Interesting. Can you share some details about your setup? What interconnect did you use, 1GB ethernet, 10GB ethernet, or inifiniband? How was performance and stability now? Can you get close to 100% disk I/O capacity, or is it mainly limited by other factors (network, CPU, etc.) ? -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Stratton
2009-May-31 15:19 UTC
Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen
On Sun, 31 May 2009, Fajar A. Nugraha wrote:> On Sun, May 31, 2009 at 8:49 PM, Nathan Stratton <nathan@robotics.net> wrote: >> Some of that is a bit dated, we have been running DRBD in 4 production 10 TB >> clusters for over a year now. > > Interesting. Can you share some details about your setup? > > What interconnect did you use, 1GB ethernet, 10GB ethernet, or inifiniband? > How was performance and stability now? Can you get close to 100% disk > I/O capacity, or is it mainly limited by other factors (network, CPU, > etc.) ?Infiniband and we get about 350 MB/s, that not raw, but better then other redundant setups we tried. -Nathan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Eisenberg
2009-May-31 19:13 UTC
RE: [Xen-users] Advice on redundant SAN/NAS storage for Xen
As Fajar said, fencing is a major concern. I''ve used DRBD successfully for such setups, but it is the sort of thing you''ll want to aggressively and thoroughly test prior to deployment, because you don''t ever want to deal with DRBD split brain. :) Best Regards Nathan Eisenberg Sr. Systems Administrator Atlas Networks, LLC support@atlasnetworks.us http://support.atlasnetworks.us/portal -----Original Message----- From: Rudi Ahlers [mailto:rudiahlers@gmail.com] Sent: Saturday, May 30, 2009 10:50 PM To: Nathan Eisenberg Cc: Xen User-List Subject: Re: [Xen-users] Advice on redundant SAN/NAS storage for Xen On Sat, May 30, 2009 at 10:19 PM, Nathan Eisenberg <nathan@atlasnetworks.us> wrote:> In my experience, effectively using DRBD is a major undertaking, and it can VERY easily cause you a lot of heartache. Approach with caution! > > Best Regards > Nathan Eisenberg > Sr. Systems Administrator > Atlas Networks, LLC > support@atlasnetworks.us > http://support.atlasnetworks.us/portalHi Nathan, Can you please elaborate on this? And my question, on this same topic, would be, what is wrong with using 2 standard servers (i.e. not a dedicated, expensive NAS device) with 2 or 4 HDD''s each, setup with something like DRBD / RedHat Cluster Suite to offer one single redundant NAS / SAN server? Or, even using something like http://www.openfiler.com/ or http://www.freenas.org/ ? -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users