Hallo list, hope that so can help me on this topic. I''d like to know where the *real* advantages of Nexenta/ZFS (i.e. ZFS/StorageTek) over DRBD/Heartbeat are. I''m pretty new to this topic and hence do not have enough experience to judge their respective advantages/disadvantages reasonably. Any suggestion would be appreciated. -- Best regards Axel Schmalowsky Platform Engineer ___________________________________ domainfactory GmbH Oskar-Messter-Str. 33 85737 Ismaning Germany Telefon: +49 (0)89 / 55266-356 Telefax: +49 (0)89 / 55266-222 E-Mail: aschmalowsky at df.eu Internet: www.df.eu Registergericht: Amtsgericht M?nchen HRB 150294, Gesch?ftsf?hrer Tobias Marburg, Jochen Tuchbreiter
Well, obviously - its Linux vs. OpenSolaris question. Most serious advantage of OpenSolaris is ZFS and its enterprise level storage stack. Linux just not there yet.. On Wed, 2008-09-10 at 14:51 +0200, Axel Schmalowsky wrote:> Hallo list, > > hope that so can help me on this topic. > > I''d like to know where the *real* advantages of Nexenta/ZFS (i.e. ZFS/StorageTek) over DRBD/Heartbeat are. > I''m pretty new to this topic and hence do not have enough experience to judge their respective advantages/disadvantages reasonably. > > Any suggestion would be appreciated. > >
>I''d like to know where the *real* advantages of Nexenta/ZFS (i.e. >ZFS/StorageTek) over DRBD/Heartbeat are.The main advantage of OpenSolaris is native ZFS, the many advantages of which are well described in many places, such as http://opensolaris.org/os/community/zfs/docs/zfs_last.pdf. A disadvantage, however, is that Sun StorageTek Availability Suite (AVS), the DRBD equivalent in OpenSolaris, is much less flexible than DRBD. For example, AVS is intended to replicate in one direction, from a primary to a secondary, whereas DRBD can switch on the fly. See http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 for details on this. -- Maurice Volaski, mvolaski at aecom.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University
On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote:> A disadvantage, however, is that Sun StorageTek Availability Suite > (AVS), the DRBD equivalent in OpenSolaris, is much less flexible than > DRBD. For example, AVS is intended to replicate in one direction, > from a primary to a secondary, whereas DRBD can switch on the fly. > See > http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 > for details on this.I would be curious to see production environments "switching" direction on the fly at that low level... Usually some top-level brain does that in context of HA fail-over and so on. well, AVS actually does reverse synchronization and does it very good.
>On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: >> A disadvantage, however, is that Sun StorageTek Availability Suite >> (AVS), the DRBD equivalent in OpenSolaris, is much less flexible than >> DRBD. For example, AVS is intended to replicate in one direction, >> from a primary to a secondary, whereas DRBD can switch on the fly. >> See >> http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 >> for details on this. > >I would be curious to see production environments "switching" direction >on the fly at that low level... Usually some top-level brain does that >in context of HA fail-over and so on.By switching on the fly, I mean if the primary services are taken down and then brought up on the secondary, the direction of synchronization gets reversed. That''s not possible with AVS because...>well, AVS actually does reverse synchronization and does it very good.It''s a one-time operation that "re-reverses" once it completes. -- Maurice Volaski, mvolaski at aecom.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University
On Wed, 2008-09-10 at 15:00 -0400, Maurice Volaski wrote:> >On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: > >> A disadvantage, however, is that Sun StorageTek Availability Suite > >> (AVS), the DRBD equivalent in OpenSolaris, is much less flexible than > >> DRBD. For example, AVS is intended to replicate in one direction, > >> from a primary to a secondary, whereas DRBD can switch on the fly. > >> See > >> http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 > >> for details on this. > > > >I would be curious to see production environments "switching" direction > >on the fly at that low level... Usually some top-level brain does that > >in context of HA fail-over and so on. > > By switching on the fly, I mean if the primary services are taken > down and then brought up on the secondary, the direction of > synchronization gets reversed. That''s not possible with AVS because... > > >well, AVS actually does reverse synchronization and does it very good. > > It''s a one-time operation that "re-reverses" once it completes.When primary is repaired you want to have it on-line and retain the changes made on the secondary. Your secondary did the job and switched back to its secondary role. This HA fail-back cycle could be repeated as many times as you need using reverse sync command.
>On Wed, 2008-09-10 at 15:00 -0400, Maurice Volaski wrote: >> >On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: >> >> A disadvantage, however, is that Sun StorageTek Availability Suite >> >> (AVS), the DRBD equivalent in OpenSolaris, is much less flexible than >> >> DRBD. For example, AVS is intended to replicate in one direction, >> >> from a primary to a secondary, whereas DRBD can switch on the fly. >> >> See >> >> http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 >> >> for details on this. >> > >> >I would be curious to see production environments "switching" direction >> >on the fly at that low level... Usually some top-level brain does that >> >in context of HA fail-over and so on. >> >> By switching on the fly, I mean if the primary services are taken >> down and then brought up on the secondary, the direction of >> synchronization gets reversed. That''s not possible with AVS because... >> >> >well, AVS actually does reverse synchronization and does it very good. >> >> It''s a one-time operation that "re-reverses" once it completes. > >When primary is repaired you want to have it on-line and retain the >changes made on the secondary.Not necessarily. Even when the primary is ready to go back into service, I may not want to revert to it for one reason or another. That means I am without a live mirror because AVS'' realtime mirroring is only one direction, primary to secondary. -- Maurice Volaski, mvolaski at aecom.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University
On Wed, 2008-09-10 at 18:37 -0400, Maurice Volaski wrote:> >On Wed, 2008-09-10 at 15:00 -0400, Maurice Volaski wrote: > >> >On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: > >> >> A disadvantage, however, is that Sun StorageTek Availability Suite > >> >> (AVS), the DRBD equivalent in OpenSolaris, is much less flexible than > >> >> DRBD. For example, AVS is intended to replicate in one direction, > >> >> from a primary to a secondary, whereas DRBD can switch on the fly. > >> >> See > >> >> http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 > >> >> for details on this. > >> > > >> >I would be curious to see production environments "switching" direction > >> >on the fly at that low level... Usually some top-level brain does that > >> >in context of HA fail-over and so on. > >> > >> By switching on the fly, I mean if the primary services are taken > >> down and then brought up on the secondary, the direction of > >> synchronization gets reversed. That''s not possible with AVS because... > >> > >> >well, AVS actually does reverse synchronization and does it very good. > >> > >> It''s a one-time operation that "re-reverses" once it completes. > > > >When primary is repaired you want to have it on-line and retain the > >changes made on the secondary. > > Not necessarily. Even when the primary is ready to go back into > service, I may not want to revert to it for one reason or another. > That means I am without a live mirror because AVS'' realtime mirroring > is only one direction, primary to secondary.This why I tried to state that this is not realistic environment for non-shared storage HA deployments. DRBD trying to emulate shared-storage behavior at a wrong level where in fact usage of FC/iSCSI-connected storage needs to be considered.
>On Wed, 2008-09-10 at 18:37 -0400, Maurice Volaski wrote: >> >On Wed, 2008-09-10 at 15:00 -0400, Maurice Volaski wrote: >> >> >On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: >> >> >> A disadvantage, however, is that Sun StorageTek Availability Suite >> >> >> (AVS), the DRBD equivalent in OpenSolaris, is much less >>flexible than >> >> >> DRBD. For example, AVS is intended to replicate in one direction, >> >> >> from a primary to a secondary, whereas DRBD can switch on the fly. >> >> >> See >> >> >> http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 >> >> >> for details on this. >> >> > >> >> >I would be curious to see production environments "switching" direction >> >> >on the fly at that low level... Usually some top-level brain does that >> >> >in context of HA fail-over and so on. >> >> >> >> By switching on the fly, I mean if the primary services are taken >> >> down and then brought up on the secondary, the direction of >> >> synchronization gets reversed. That''s not possible with AVS because... >> >> >> >> >well, AVS actually does reverse synchronization and does it very good. >> >> >> >> It''s a one-time operation that "re-reverses" once it completes. >> > >> >When primary is repaired you want to have it on-line and retain the >> >changes made on the secondary. >> >> Not necessarily. Even when the primary is ready to go back into >> service, I may not want to revert to it for one reason or another. >> That means I am without a live mirror because AVS'' realtime mirroring >> is only one direction, primary to secondary. > >This why I tried to state that this is not realistic environment for >non-shared storage HA deployments.What''s not realistic? DRBD''s highly flexible ability to switch roles on the fly is a huge advantage over AVS. But this is not to say AVS is not realistic. It''s just a limitation.>DRBD trying to emulate shared-storage >behavior at a wrong level where in fact usage of FC/iSCSI-connected >storage needs to be considered.This makes no sense to me. We''re talking about mirroring the storage of two physical and independent systems. How did the concept of "shared storage" get in here? -- Maurice Volaski, mvolaski at aecom.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University
On Wed, 2008-09-10 at 19:10 -0400, Maurice Volaski wrote:> >On Wed, 2008-09-10 at 18:37 -0400, Maurice Volaski wrote: > >> >On Wed, 2008-09-10 at 15:00 -0400, Maurice Volaski wrote: > >> >> >On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: > >> >> >> A disadvantage, however, is that Sun StorageTek Availability Suite > >> >> >> (AVS), the DRBD equivalent in OpenSolaris, is much less > >>flexible than > >> >> >> DRBD. For example, AVS is intended to replicate in one direction, > >> >> >> from a primary to a secondary, whereas DRBD can switch on the fly. > >> >> >> See > >> >> >> http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 > >> >> >> for details on this. > >> >> > > >> >> >I would be curious to see production environments "switching" direction > >> >> >on the fly at that low level... Usually some top-level brain does that > >> >> >in context of HA fail-over and so on. > >> >> > >> >> By switching on the fly, I mean if the primary services are taken > >> >> down and then brought up on the secondary, the direction of > >> >> synchronization gets reversed. That''s not possible with AVS because... > >> >> > >> >> >well, AVS actually does reverse synchronization and does it very good. > >> >> > >> >> It''s a one-time operation that "re-reverses" once it completes. > >> > > >> >When primary is repaired you want to have it on-line and retain the > >> >changes made on the secondary. > >> > >> Not necessarily. Even when the primary is ready to go back into > >> service, I may not want to revert to it for one reason or another. > >> That means I am without a live mirror because AVS'' realtime mirroring > >> is only one direction, primary to secondary. > > > >This why I tried to state that this is not realistic environment for > >non-shared storage HA deployments. > > What''s not realistic? DRBD''s highly flexible ability to switch roles > on the fly is a huge advantage over AVS. But this is not to say AVS > is not realistic. It''s just a limitation. > > >DRBD trying to emulate shared-storage > >behavior at a wrong level where in fact usage of FC/iSCSI-connected > >storage needs to be considered. > > This makes no sense to me. We''re talking about mirroring the storage > of two physical and independent systems. How did the concept of > "shared storage" get in here?This is really outside of ZFS discussion now... But your point taken. If you want mirror-like behavior of your 2-node cluster, you''ll get some benefits of DRBD but my point is that such solution trying to solve two problems at the same time: replication and availability, which is in my opinion plain wrong.
>On Wed, 2008-09-10 at 19:10 -0400, Maurice Volaski wrote: >> >On Wed, 2008-09-10 at 18:37 -0400, Maurice Volaski wrote: >> >> >On Wed, 2008-09-10 at 15:00 -0400, Maurice Volaski wrote: >> >> >> >On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: >> >> >> >> A disadvantage, however, is that Sun StorageTek >>Availability Suite >> >> >> >> (AVS), the DRBD equivalent in OpenSolaris, is much less >> >>flexible than >> >> >> >> DRBD. For example, AVS is intended to replicate in one >>direction, >> >> >> >> from a primary to a secondary, whereas DRBD can switch >>on the fly. >> >> >> >> See >> >> >> >> >>http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 >> >> >> >> for details on this. >> >> >> > >> >> >> >I would be curious to see production environments >>"switching" direction >> >> >> >on the fly at that low level... Usually some top-level >>brain does that >> >> >> >in context of HA fail-over and so on. >> >> >> >> >> >> By switching on the fly, I mean if the primary services are taken >> >> >> down and then brought up on the secondary, the direction of >> >> >> synchronization gets reversed. That''s not possible with >>AVS because... >> >> >> >> >> >> >well, AVS actually does reverse synchronization and does >>it very good. >> >> >> >> >> >> It''s a one-time operation that "re-reverses" once it completes. >> >> > >> >> >When primary is repaired you want to have it on-line and retain the >> >> >changes made on the secondary. >> >> >> >> Not necessarily. Even when the primary is ready to go back into >> >> service, I may not want to revert to it for one reason or another. >> >> That means I am without a live mirror because AVS'' realtime mirroring >> >> is only one direction, primary to secondary. >> > >> >This why I tried to state that this is not realistic environment for >> >non-shared storage HA deployments. >> >> What''s not realistic? DRBD''s highly flexible ability to switch roles >> on the fly is a huge advantage over AVS. But this is not to say AVS >> is not realistic. It''s just a limitation. >> >> >DRBD trying to emulate shared-storage >> >behavior at a wrong level where in fact usage of FC/iSCSI-connected >> >storage needs to be considered. >> >> This makes no sense to me. We''re talking about mirroring the storage >> of two physical and independent systems. How did the concept of >> "shared storage" get in here? > >This is really outside of ZFS discussion now... But your point taken. If >you want mirror-like behavior of your 2-node cluster, you''ll get some >benefits of DRBD but my point is that such solution trying to solve two >problems at the same time: replication and availability, which is in my >opinion plain wrong.Uh, no, DRBD addresses only replication. Linux-HA (aka Heartbeat) address availability. They can be an integrated solution and are to some degree intended that way, so I have no idea where your opinion is coming from. For replication, OpenSolaris is largely limited to using AVS, whose functionality is limited, at least relative to DRBD. But there seems to be a few options to implement availability, which should include Linux-HA itself as it should run on OpenSolaris! But relevant to the poster''s initial question, ZFS is so far and away more advanced than any Linux filesystem can even dream about that it handily nullifies any disadvantage in having to run AVS. -- Maurice Volaski, mvolaski at aecom.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University
On Wed, 2008-09-10 at 19:42 -0400, Maurice Volaski wrote:> >On Wed, 2008-09-10 at 19:10 -0400, Maurice Volaski wrote: > >> >On Wed, 2008-09-10 at 18:37 -0400, Maurice Volaski wrote: > >> >> >On Wed, 2008-09-10 at 15:00 -0400, Maurice Volaski wrote: > >> >> >> >On Wed, 2008-09-10 at 14:36 -0400, Maurice Volaski wrote: > >> >> >> >> A disadvantage, however, is that Sun StorageTek > >>Availability Suite > >> >> >> >> (AVS), the DRBD equivalent in OpenSolaris, is much less > >> >>flexible than > >> >> >> >> DRBD. For example, AVS is intended to replicate in one > >>direction, > >> >> >> >> from a primary to a secondary, whereas DRBD can switch > >>on the fly. > >> >> >> >> See > >> >> >> >> > >>http://www.opensolaris.org/jive/thread.jspa?threadID=68881&tstart=30 > >> >> >> >> for details on this. > >> >> >> > > >> >> >> >I would be curious to see production environments > >>"switching" direction > >> >> >> >on the fly at that low level... Usually some top-level > >>brain does that > >> >> >> >in context of HA fail-over and so on. > >> >> >> > >> >> >> By switching on the fly, I mean if the primary services are taken > >> >> >> down and then brought up on the secondary, the direction of > >> >> >> synchronization gets reversed. That''s not possible with > >>AVS because... > >> >> >> > >> >> >> >well, AVS actually does reverse synchronization and does > >>it very good. > >> >> >> > >> >> >> It''s a one-time operation that "re-reverses" once it completes. > >> >> > > >> >> >When primary is repaired you want to have it on-line and retain the > >> >> >changes made on the secondary. > >> >> > >> >> Not necessarily. Even when the primary is ready to go back into > >> >> service, I may not want to revert to it for one reason or another. > >> >> That means I am without a live mirror because AVS'' realtime mirroring > >> >> is only one direction, primary to secondary. > >> > > >> >This why I tried to state that this is not realistic environment for > >> >non-shared storage HA deployments. > >> > >> What''s not realistic? DRBD''s highly flexible ability to switch roles > >> on the fly is a huge advantage over AVS. But this is not to say AVS > >> is not realistic. It''s just a limitation. > >> > >> >DRBD trying to emulate shared-storage > >> >behavior at a wrong level where in fact usage of FC/iSCSI-connected > >> >storage needs to be considered. > >> > >> This makes no sense to me. We''re talking about mirroring the storage > >> of two physical and independent systems. How did the concept of > >> "shared storage" get in here? > > > >This is really outside of ZFS discussion now... But your point taken. If > >you want mirror-like behavior of your 2-node cluster, you''ll get some > >benefits of DRBD but my point is that such solution trying to solve two > >problems at the same time: replication and availability, which is in my > >opinion plain wrong. > > Uh, no, DRBD addresses only replication. Linux-HA (aka Heartbeat) > address availability. They can be an integrated solution and are to > some degree intended that way, so I have no idea where your opinion > is coming from.Because in my opinion DRBD takes some responsibility of management layer if you will. Classic, predominant replication in HA clusters schema is primary-backup (or master-slave) and backup by definition is not necessary primary-identical system. Having said that, it is noble for DRBD to implement role switching and not a bad idea for many small deployments.> For replication, OpenSolaris is largely limited to using AVS, whose > functionality is limited, at least relative to DRBD. But there seems > to be a few options to implement availability, which should include > Linux-HA itself as it should run on OpenSolaris!Everything is implementable and I believe AVS designers thought about dynamic switching of roles, but they end up with what we have today, they likely discarded this idea. AVS not switching roles and forces IT admins to use it as primary-backup data protection service only.> But relevant to the poster''s initial question, ZFS is so far and away > more advanced than any Linux filesystem can even dream about that it > handily nullifies any disadvantage in having to run AVS.Right.
Erast Benson wrote:>> Uh, no, DRBD addresses only replication. Linux-HA (aka Heartbeat) >> address availability. They can be an integrated solution and are to >> some degree intended that way, so I have no idea where your opinion >> is coming from. >> > > Because in my opinion DRBD takes some responsibility of management layer > if you will. Classic, predominant replication in HA clusters schema is > primary-backup (or master-slave) and backup by definition is not > necessary primary-identical system. Having said that, it is noble for > DRBD to implement role switching and not a bad idea for many small > deployments. >The problem with fully automated systems for remote replication is that they are fully automated. This opens you up to a set of failure modes that you may want to avoid, such as replication of data that you don''t want to replicate. This is why most replication is used to support disaster recovery cases and the procedures wrapped around disaster recovery also consider the case where the primary data has been damaged -- and you really don''t want that damage to spread. It so happens that snapshots are another method which can be used to limit the spread of damage, so there might be an opportunity here. By analogy, in the Oracle world, RAC does not replace DataGuard.>> For replication, OpenSolaris is largely limited to using AVS, whose >> functionality is limited, at least relative to DRBD. But there seems >> to be a few options to implement availability, which should include >> Linux-HA itself as it should run on OpenSolaris! >>I disagree, there are many ways to remotely replicate Solaris systems. TrueCopy and SRDF are perhaps the most popular, but almost all storage arrays have some sort of system. In truth, there is little market demand for fully automated solutions at the OS level because of the reasons mentioned above.> > Everything is implementable and I believe AVS designers thought about > dynamic switching of roles, but they end up with what we have today, > they likely discarded this idea. >It is open source... -- richard
Let me drag this thread kicking and screaming back to ZFS... Use case: - We need an NFS server that can be replicated to another building to handle both scheduled powerdowns and unplanned outages. For scheduled powerdowns we''d want to fail over a week in advance, and fail back some time later. - We will use a virtual IP for seamless failover, and require that we not get stale NFS filehandles during a failover event, as world reboots are messy and expensive. - We do _not_ require synchronous replication. Async is fine. Today, there is _no_ rasonably priced solution that allows us to use ZFS for this. Yes, we have SRDF, and use it where required, but it''s hideously expensive, and has its own risks (see below). Today our solution for the above is NetApp filers. We''re not thrilled with everything about them, but they mostly work. One of the projects I''ve done is set up automated zfs replication for pure DR. But sadly this is of limited use, as Solaris / ZFS''s lack of ability to maintain NFS file handles across systems is a deal killer for most of our users. AVS does not appear to support our weeks-long failover model, and exposes us to too much risk of simultaneous data loss. SRDF and Veritas replication have the same data loss risk. SRDF can easily support weeks-long a personality swap, I don''t know enough about Veritas'' product. -- Carson
>The problem with fully automated systems for remote replication is >that they are fully automated. This opens you up to a set of failure modes >that you may want to avoid, such as replication of data that you don''t >want to replicate. This is why most replication is used to support disaster >recovery cases and the procedures wrapped around disaster recovery >also consider the case where the primary data has been damaged -- and >you really don''t want that damage to spread.I seem to be misrepresenting how automatic DRBD is because it offers a rich set of policies and strategies to deal with faults. For one thing, it (along with heartbeat) will bend over backwards to avoid disasters such as split-brain. It''s more likely that one would get split-brain by manually executing the wrong command than by DRBD''s (or heartbeat''s) doing.>I disagree, there are many ways to remotely replicate Solaris systems. >TrueCopy and SRDF are perhaps the most popular, but almost allI was referring to cost free options. -- Maurice Volaski, mvolaski at aecom.yu.edu Computing Support, Rose F. Kennedy Center Albert Einstein College of Medicine of Yeshiva University
Carson Gaspar wrote:> Let me drag this thread kicking and screaming back to ZFS... > > Use case: > > - We need an NFS server that can be replicated to another building to > handle both scheduled powerdowns and unplanned outages. For scheduled > powerdowns we''d want to fail over a week in advance, and fail back some > time later. >For campus or metro sized systems, many people just use HA clusters. The complexity level is similar and you automatically avoid the NFS file handle problem. There is a lot of expertise in this area as NFS is one of the most popular clustered services. http://www.opensolaris.org/os/community/ha-clusters/ -- richard
Richard Elling wrote:> > For campus or metro sized systems, many people just use HA clusters. > The complexity level is similar and you automatically avoid the NFS > file handle problem. There is a lot of expertise in this area as NFS > is one of the most popular clustered services. > http://www.opensolaris.org/os/community/ha-clusters/What handles the data replication in this case? It _looks_ like we''re back to SRDF / AVS / Veritas, unless there''s some magic I''m missing. Which, as I stated in my original email, all have data loss issues (replicating errors at the block level across all hosts). -- Carson
Carson Gaspar wrote:> Richard Elling wrote: > >> For campus or metro sized systems, many people just use HA clusters. >> The complexity level is similar and you automatically avoid the NFS >> file handle problem. There is a lot of expertise in this area as NFS >> is one of the most popular clustered services. >> http://www.opensolaris.org/os/community/ha-clusters/ >> > > What handles the data replication in this case?Mirroring. KISS. -- richard> It _looks_ like we''re > back to SRDF / AVS / Veritas, unless there''s some magic I''m missing. > Which, as I stated in my original email, all have data loss issues > (replicating errors at the block level across all hosts). > >