>>> On 2009/11/26 at 02:26, Peter Braun <xenware@gmail.com> wrote: > Hi, > > its not my document.I don''t think Fajar was suggesting that it was your document, or blaming you for what he saw as a couple of possible problems in configuration described in the document. He was just pointing out that there are weaknesses with manual fencing that can lead to corruption of your file system, so, when using GFS, you need to set up real fencing devices to avoid major filesystem problems.> > Actually Ive tried to install H/A SAN according do this document but > without success. > > Am looking for opensource solution of H/A SAN - and this is close to my goal. > > > The basics: > > 1) 2 machines with some HDD space synchronized between them with DRBD.OK so far...or, if you can afford some sort of storage on the back end capable of being presented to two hosts simultaneously (like FC-based storage), you can use that, too.> > 2) Now am little bit confused what shall I use above the DRBD block > device? > - some cluster FS like OCFS2 or GFS? > - LVM?Your "SAN controllers" need not actually have a filesystem on them, unless you''re trying to run DRBD on the same systems that are running your domUs. If you have separate machines doing all of the SAN functionality, simply synchronize with DRBD and present with iSCSI. Then you can use a CRM tool (like Heartbeat or one of the others out there) to manage switching IP addresses between your redundant SAN heads. At this stage you need not worry about OCFS2 or GFS. On your Xen servers, use an iSCSI client or iSCSI HBA to connect to the IP address(es).> > 3) create files with DD on cluster FS and export them with iSCSI? > create LVM partitions and export them with iSCSI?This is where you choose between a cluster-aware FS and a cluster-aware volume manager on the servers running Xen (or both). Either one should be fine - I use a cluster-aware FS (OCFS2) and use files for each of the domU disks. One thing to note with a cluster-aware volume manager, if you go this route you still need to figure out a way to synchronize your domU configurations between the machines. This may mean using a tool like rsync, or creating a volume on your volume manager that''s just for configuration files and using a cluster-aware FS on that volume to mount it on all of the Xen servers.> > 4) how to make iSCSI target highly available? > - configure iSCSI on virtual IP/another IP and run it as HA service > - configure separate iSCSI targets on both SAN hosts and > connect it to Xen server as multipath?I think either one of these should work, though you need to make sure that the later of the two options is okay given the iSCSI Target Daemon that you use - with buffering and caching you may run into some issues with writes not being committed to the disk in the proper order, which could lead to corruption.> > 5) hearbeat configurationYep...on the SAN controllers for the IPs and iSCSI targets, and possibly on the Xen servers, as well, for the cluster-aware FS or the cluster-aware volume manager.> > VM machines with iSCSI HDD space on SAN should survive reboot/non > availability of one SAN hosts without interruption nor noticing that > SAN is degraded. > > Is that even possible?Absolutely - the primary issue is getting that failover set up correctly such that the small interruption to iSCSI service that your Xen servers will experience does not cause them to think the target has gone away or cause them to miss iSCSI disk writes. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hi i am looking at all this discussion about ha storage. it looks to me like there are a lot of complicated solutions for the problem. i guess that it will be not very complicated to do all this stuff in the virtual disc driver of xen. basicly its just double all the read and write access to the disc image. and do the resync stuff there. meight be we shold ask the developper team if they can do it. the question is ... are there some other people that also want a easy solution or its just me. roland On Thu, 26 Nov 2009 16:29:43 +0100, Nick Couchman <Nick.Couchman@seakr.com> wrote:>>>> On 2009/11/26 at 02:26, Peter Braun <xenware@gmail.com> wrote: >> Hi, >> >> its not my document. > > I don''t think Fajar was suggesting that it was your document, or blaming > you for what he saw as a couple of possible problems in configuration > described in the document. He was just pointing out that there are > weaknesses with manual fencing that can lead to corruption of your file > system, so, when using GFS, you need to set up real fencing devices to > avoid major filesystem problems. > >> >> Actually Ive tried to install H/A SAN according do this document but >> without success. >> >> Am looking for opensource solution of H/A SAN - and this is close to my >> goal. >> >> >> The basics: >> >> 1) 2 machines with some HDD space synchronized between them with >> DRBD. > > OK so far...or, if you can afford some sort of storage on the back end > capable of being presented to two hosts simultaneously (like FC-based > storage), you can use that, too. > >> >> 2) Now am little bit confused what shall I use above the DRBD block >> device? >> - some cluster FS like OCFS2 or GFS? >> - LVM? > > Your "SAN controllers" need not actually have a filesystem on them, > unless you''re trying to run DRBD on the same systems that are running > your domUs. If you have separate machines doing all of the SAN > functionality, simply synchronize with DRBD and present with iSCSI. > Then you can use a CRM tool (like Heartbeat or one of the others out > there) to manage switching IP addresses between your redundant SAN > heads. At this stage you need not worry about OCFS2 or GFS. On your > Xen servers, use an iSCSI client or iSCSI HBA to connect to the IP > address(es). > >> >> 3) create files with DD on cluster FS and export them with iSCSI? >> create LVM partitions and export them with iSCSI? > > This is where you choose between a cluster-aware FS and a cluster-aware > volume manager on the servers running Xen (or both). Either one should > be fine - I use a cluster-aware FS (OCFS2) and use files for each of the > domU disks. One thing to note with a cluster-aware volume manager, if > you go this route you still need to figure out a way to synchronize your > domU configurations between the machines. This may mean using a tool > like rsync, or creating a volume on your volume manager that''s just for > configuration files and using a cluster-aware FS on that volume to mount > it on all of the Xen servers. > >> >> 4) how to make iSCSI target highly available? >> - configure iSCSI on virtual IP/another IP and run it as HA >> service >> - configure separate iSCSI targets on both SAN hosts and >> connect it to Xen server as multipath? > > I think either one of these should work, though you need to make sure > that the later of the two options is okay given the iSCSI Target Daemon > that you use - with buffering and caching you may run into some issues > with writes not being committed to the disk in the proper order, which > could lead to corruption. > >> >> 5) hearbeat configuration > > Yep...on the SAN controllers for the IPs and iSCSI targets, and possibly > on the Xen servers, as well, for the cluster-aware FS or the > cluster-aware volume manager. > >> >> VM machines with iSCSI HDD space on SAN should survive reboot/non >> availability of one SAN hosts without interruption nor noticing that >> SAN is degraded. >> >> Is that even possible? > > Absolutely - the primary issue is getting that failover set up correctly > such that the small interruption to iSCSI service that your Xen servers > will experience does not cause them to think the target has gone away or > cause them to miss iSCSI disk writes. > > -Nick > > > > -------- > > This e-mail may contain confidential and privileged material for the > sole use of the intended recipient. If this email is not intended for > you, or you are not responsible for the delivery of this message to the > intended recipient, please note that this message may contain SEAKR > Engineering (SEAKR) Privileged/Proprietary Information. In such a case, > you are strictly prohibited from downloading, photocopying, distributing > or otherwise using this message, its contents or attachments in any > way. If you have received this message in error, please notify us > immediately by replying to this e-mail and delete the message from your > mailbox. Information contained in this message that does not relate to > the business of SEAKR is neither endorsed by nor attributable to SEAKR._______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, 2009-11-26 at 16:45 +0100, Roland Frei wrote:> hi > > i am looking at all this discussion about ha storage. > it looks to me like there are a lot of complicated solutions for the > problem. > > i guess that it will be not very complicated to do all this stuff in > the > virtual disc driver of xen. > basicly its just double all the read and write access to the disc > image. > and do the resync stuff there. > > meight be we shold ask the developper team if they can do it. > > the question is ... are there some other people that also want a > easy > solution or its just me. > > rolandThere are complicated solutions because it is a complicated setup. Where''s the fun in everything being done for you? Setting up a clustered or replicated SAN is not an easy task for beginners, you may want to read a bit about each technology. Or set up test servers that aren''t critical machines and play around with it. The easiest thing would be to setup the cluster using DRBD and OCFS2, mounting it on your Xen host and using image files on the hard drive. In this case your images can potentially survive a Xen host failure (if there is no corruption). Not sure if posting links is allowed, but here is a howto on creating an iSCSI SAN with DRBD, OCFS2 and Heartbeat. http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/ If followed correctly, it will give you a storage platform to create your VM image files on. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Nov 26, 2009 at 10:45 PM, Roland Frei <frei@datatrust.ch> wrote:> i guess that it will be not very complicated to do all this stuff in the > virtual disc driver of xen. > basicly its just double all the read and write access to the disc image. and > do the resync stuff there.If double write is what you need, my first suggestion already does that: - san exports its storage as iscsi - server/PC imports iscsi share from both san - setup raid1 on server using those two iscsi imports no need to modify xen code as linux md can already do it. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> If double write is what you need, my first suggestion already does > that: > - san exports its storage as iscsi > - server/PC imports iscsi share from both san > - setup raid1 on server using those two iscsi imports > > no need to modify xen code as linux md can already do it. > > -- > FajarThe problem I''ve encountered with doing HA storage in the domU is that you are now doubling the required throughput of the dom0''s iSCSI interface, which is already easy to bottleneck if it''s only GigE. 125MB/s may seem like a lot of throughput, but if you start by cutting that in half for RAID traffic, and then share that remaining bandwidth across 5 domU''s, you''re going to fill it up pretty quick. The other problem is that if you ever take a node down, and you have 5 domUs configured in this way, when that node comes back up, all of your domUs will attempt to resync the mirror simultaneously. I''d bet that this could actually lead to catastrophic kernel panics, as the domUs can become IO starved. -Nathan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > The problem I''ve encountered with doing HA storage in the domU is thatyou are> now doubling the required throughput of the dom0''s iSCSI interface,which is> already easy to bottleneck if it''s only GigE. 125MB/s may seem like alot of> throughput, but if you start by cutting that in half for RAID traffic,and> then share that remaining bandwidth across 5 domU''s, you''re going tofill it> up pretty quick.I think that you wouldn''t run both SAN links across a single GigE - that would kind of defeat the purpose. I agree with everything else you''ve said though. DRBD seems to solve all the problems presented here, unless someone knows something I don''t. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users