Is it possible to use Lustre to make a single filesystem that has a mirror between two locations such that if the power were to be lost in one location that the other location could continue to run uninterrupted? If not, is anyone aware of solution that would allow this? I would really like to have an active/active cluster so that I could run an apache cluster on top of the filesystem, and load balance all Web activity across any of the apache nodes which read/write to storage which is synchronously mirrored between two storage systems that are a couple of kilometeres apart. Thanks for any reference, experience or pointers to resources. -- Evan Rempel
On Fri, 2009-08-07 at 11:18 -0700, Evan Rempel wrote:> Is it possible to use Lustre to make a single filesystem > that has a mirror between two locations such that if > the power were to be lost in one location that the other > location could continue to run uninterrupted?Mirror or replication? There is a difference. Mirror implies that the two sides are coherent, meaning that a write on one side cannot be considered complete until the other side is written too. Replication is lazy mirroring with some limitations. The major limitation is that because it is lazy, you do not have a guarantee that just because a file is on one side, it is available on the other side. It will only become available to the other side when resource constraints (such as bandwidth and latency) allow it to be available. If you have lots of bandwidth and low latency, the delay could be quite short. The other thing is that I don''t know that our replication feature is going to be bi-directional. That is, I don''t know that you can have to active filesystems replicating to each other.> I would really like to have an active/active cluster so > that I could run an apache cluster on top > of the filesystem, and load balance all Web activity > across any of the apache nodes which read/write to > storage which is synchronously mirrored between > two storage systems that are a couple of kilometeres > apart.This is a difficult problem to solve. As long as each side is writing to different files or parts of the filesystem, it''s easy, but what happens when both sides open the same file for a write operation? Last write wins? That''s not usually acceptable. Perhaps one of the engineers working on our replication feature will chime in on whether replication can be bi-directional or not. Just given the complexity of the conflict problem, I tend to doubt it. It might be that the replication system calls out to a user defined function to resolve conflicts. But that is just a total WAG on my part. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090807/530f5bce/attachment.bin
On Aug 07, 2009 20:24 -0400, Brian J. Murrell wrote:> The other thing is that I don''t know that our replication feature is > going to be bi-directional. That is, I don''t know that you can have to > active filesystems replicating to each other.No, the replication feature in 2.0 will strictly be one directional data copying. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
How does this replication feature compare to rsync in performance and ease of use? On Sat, Aug 8, 2009 at 2:42 AM, Andreas Dilger<adilger at sun.com> wrote:> On Aug 07, 2009 ?20:24 -0400, Brian J. Murrell wrote: >> The other thing is that I don''t know that our replication feature is >> going to be bi-directional. ?That is, I don''t know that you can have to >> active filesystems replicating to each other. > > No, the replication feature in 2.0 will strictly be one directional > data copying. > > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On Tue, 2009-08-11 at 07:35 -0400, Mag Gam wrote:> How does this replication feature compare to rsync in performance and > ease of use?I don''t know the details of our replication feature, but I understand the concepts at a high level. The major difference between the replication feature and rsync will be that the replication service will be "fed" the changes on the filesystem by Lustre rather than it having to periodically scan the entire filesystem trying to find the changes. That''s a huge win in terms of both time and resources. Rsync becomes unusable with very large filesystem due to both the time and memory required. It''s entirely feasible that rsync could be used as a/the transport for a/the replication feature as rsync can be given a list of files to operate on rather than scanning a filesystem. Whether that''s what our replication feature does or not, I don''t know. All of this is made possible by another upcoming feature, which is changelogs. Changelogs is a cool feature in and of itself. In fact I *really* wish I could get this same feature out of the more simple, single disk filesystems, like ext3/4 as another really useful tool for it is backups (which I suppose is just another flavour of replication, albeit with an intentional delay). b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090811/c000fb0f/attachment.bin