I''m building a campus cluster with identical storage in two locations with ZFS mirrors spanning both storage frames. Data will be mirrored using zfs. I''m looking for the best way to add log devices to this campus cluster. I am considering building a separate mirrored zpool of Flash disk that span the frames, then creating zvols to use as log devices for the data zpool. Will this work? Any other suggestions? regards, jmh -- This message posted from opensolaris.org
On 18/05/2010 15:40, John Hoogerdijk wrote:> I''m building a campus cluster with identical storage in two locations with ZFS mirrors spanning both storage frames. Data will be mirrored using zfs. I''m looking for the best way to add log devices to this campus cluster.So this is a single pool with one side of the mirror in location A and one side in location B ? Log devices can be mirrored too, so why not just put a log device in each "frame" and mirror them just like you do the "normal" pool disks. What am I missing about your setup that means that won''t work ? -- Darren J Moffat
> On 18/05/2010 15:40, John Hoogerdijk wrote: > > I''m building a campus cluster with identical > storage in two locations with ZFS mirrors spanning > both storage frames. Data will be mirrored using zfs. > I''m looking for the best way to add log devices to > this campus cluster. > So this is a single pool with one side of the mirror > in location A and > one side in location B ? > > Log devices can be mirrored too, so why not just put > a log device in > each "frame" and mirror them just like you do the > "normal" pool disks. > > What am I missing about your setup that means that > won''t work ?Yes - mirrored log devices will work. I want to share the Flash devices with more than one clustered zone/zpool (should have stated this earlier ...) hence the use of zvols. jmh -- This message posted from opensolaris.org
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of John Hoogerdijk > > I''m building a campus cluster with identical storage in two locations > with ZFS mirrors spanning both storage frames. Data will be mirrored > using zfs. I''m looking for the best way to add log devices to this > campus cluster.Either I''m crazy, or I completely miss what you''re asking. You want to have one side of a mirror attached locally, and the other side of the mirror attached ... via iscsi or something ... across the WAN? Even if you have a really fast WAN (1Gb or so) your performance is going to be terrible, and I would be very concerned about reliability. What happens if a switch reboots or crashes? Then suddenly half of the mirror isn''t available anymore (redundancy is degraded on all pairs) and ... Will it be a degraded mirror? Or will the system just hang, waiting for iscsi IO to timeout? When it comes back online, will it intelligently resilver only the parts which have changed since? Since the mirror is now broken, and local operations can happen faster than the WAN can carry them across, will the resilver ever complete, ever? I don''t know. anyway, it just doesn''t sound like a good idea to me. It sounds like something that was meant for a clustering filesystem of some kind, not particularly for ZFS. If you are adding log devices to this, I have a couple of things to say: The whole point of a log device is to accelerate sync writes, by providing nonvolatile storage which is faster than the primary storage. You''re not going to get this if any part of the log device is at the other side of a WAN. So either add a mirror of log devices locally and not across the WAN, or don''t do it at all.> I am considering building a separate mirrored zpool of Flash disk that > span the frames, then creating zvols to use as log devices for the > data zpool. Will this work? Any other suggestions?This also sounds nonsensical to me. If your primary pool devices are Flash, then there''s no point to add separate log devices. Unless you have another type of even faster nonvolatile storage.
On Tue, May 18, 2010 20:45, Edward Ned Harvey wrote:> The whole point of a log device is to accelerate sync writes, by providing > nonvolatile storage which is faster than the primary storage. You''re not > going to get this if any part of the log device is at the other side of a > WAN. So either add a mirror of log devices locally and not across the > WAN, or don''t do it at all.A good example of using distant iSCSI with close-by SSDs: http://blogs.sun.com/jkshah/entry/zfs_with_cloud_storage_and
On Tue, 18 May 2010, Edward Ned Harvey wrote:> > Either I''m crazy, or I completely miss what you''re asking. You want to have > one side of a mirror attached locally, and the other side of the mirror > attached ... via iscsi or something ... across the WAN? Even if you have a > really fast WAN (1Gb or so) your performance is going to be terrible, and I > would be very concerned about reliability. What happens if a switch reboots > or crashes? Then suddenly half of the mirror isn''t available anymore > (redundancy is degraded on all pairs) and ... Will it be a degraded mirror? > Or will the system just hang, waiting for iscsi IO to timeout? When it > comes back online, will it intelligently resilver only the parts which have > changed since? Since the mirror is now broken, and local operations can > happen faster than the WAN can carry them across, will the resilver ever > complete, ever? I don''t know.This has been accomplished successfully before. There used to be a fellow posting here (from New Zealand I think) who used distributed storage just like that. If the WAN goes away, then zfs writes will likely hang for the iSCSI timeout period (likely 3 minutes) and then continue normally once iSCSI/zfs decides that the mirror device is not available. When the WAN returns, then zfs will send only the missing updates.> The whole point of a log device is to accelerate sync writes, by providing > nonvolatile storage which is faster than the primary storage. You''re not > going to get this if any part of the log device is at the other side of a > WAN. So either add a mirror of log devices locally and not across the WAN, > or don''t do it at all.This depends on the nature of the WAN. The WAN latency may still be relatively low as compared with drive latency. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
> > From: zfs-discuss-bounces at opensolaris.org > [mailto:zfs-discuss- > > bounces at opensolaris.org] On Behalf Of John > Hoogerdijk > > > > I''m building a campus cluster with identical > storage in two locations > > with ZFS mirrors spanning both storage frames. Data > will be mirrored > > using zfs. I''m looking for the best way to add log > devices to this > > campus cluster. > > Either I''m crazy, or I completely miss what you''re > asking. You want to have > one side of a mirror attached locally, and the other > side of the mirror > attached ... via iscsi or something ... across the > WAN? Even if you have a > really fast WAN (1Gb or so) your performance is going > to be terrible, and I > would be very concerned about reliability. What > happens if a switch reboots > or crashes? Then suddenly half of the mirror isn''t > available anymore > (redundancy is degraded on all pairs) and ... Will it > be a degraded mirror? > Or will the system just hang, waiting for iscsi IO to > timeout? When it > comes back online, will it intelligently resilver > only the parts which have > changed since? Since the mirror is now broken, and > local operations can > happen faster than the WAN can carry them across, > will the resilver ever > complete, ever? I don''t know. > > anyway, it just doesn''t sound like a good idea to me. > It sounds like > omething that was meant for a clustering filesystem > of some kind, not > particularly for ZFS. > > If you are adding log devices to this, I have a > couple of things to say: > > The whole point of a log device is to accelerate sync > writes, by providing > nonvolatile storage which is faster than the primary > storage. You''re not > going to get this if any part of the log device is at > the other side of a > WAN. So either add a mirror of log devices locally > and not across the WAN, > or don''t do it at all. > > > > I am considering building a separate mirrored zpool > of Flash disk that > > span the frames, then creating zvols to use as log > devices for the > > data zpool. Will this work? Any other > suggestions? > > This also sounds nonsensical to me. If your primary > pool devices are Flash, > then there''s no point to add separate log devices. > Unless you have another > ype of even faster nonvolatile storage.Both frames are FC connected with Flash devices in the frame. Latencies are additive, so there is benefit to a logging device. The cluster is a standard HA cluster about 10km apart with identical storage in both locations, mirrored using ZFS. Think about the potential problems if I don''t mirror the log devices across the WAN.> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss >-- This message posted from opensolaris.org
> On Tue, May 18, 2010 20:45, Edward Ned Harvey wrote: > > > The whole point of a log device is to accelerate > sync writes, by providing > > nonvolatile storage which is faster than the > primary storage. You''re not > > going to get this if any part of the log device is > at the other side of a > > WAN. So either add a mirror of log devices locally > and not across the > > WAN, or don''t do it at all. > > A good example of using distant iSCSI with close-by > SSDs: > > http://blogs.sun.com/jkshah/entry/zfs_with_cloud_stora > ge_andGood stuff, but doesn''t address HA clusters and consistent storage.> > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss >-- This message posted from opensolaris.org
comment below... On May 19, 2010, at 7:50 AM, John Hoogerdijk wrote:>>> From: zfs-discuss-bounces at opensolaris.org >> [mailto:zfs-discuss- >>> bounces at opensolaris.org] On Behalf Of John >> Hoogerdijk >>> >>> I''m building a campus cluster with identical >> storage in two locations >>> with ZFS mirrors spanning both storage frames. Data >> will be mirrored >>> using zfs. I''m looking for the best way to add log >> devices to this >>> campus cluster. >> >> Either I''m crazy, or I completely miss what you''re >> asking. You want to have >> one side of a mirror attached locally, and the other >> side of the mirror >> attached ... via iscsi or something ... across the >> WAN? Even if you have a >> really fast WAN (1Gb or so) your performance is going >> to be terrible, and I >> would be very concerned about reliability. What >> happens if a switch reboots >> or crashes? Then suddenly half of the mirror isn''t >> available anymore >> (redundancy is degraded on all pairs) and ... Will it >> be a degraded mirror? >> Or will the system just hang, waiting for iscsi IO to >> timeout? When it >> comes back online, will it intelligently resilver >> only the parts which have >> changed since? Since the mirror is now broken, and >> local operations can >> happen faster than the WAN can carry them across, >> will the resilver ever >> complete, ever? I don''t know. >> >> anyway, it just doesn''t sound like a good idea to me. >> It sounds like >> omething that was meant for a clustering filesystem >> of some kind, not >> particularly for ZFS. >> >> If you are adding log devices to this, I have a >> couple of things to say: >> >> The whole point of a log device is to accelerate sync >> writes, by providing >> nonvolatile storage which is faster than the primary >> storage. You''re not >> going to get this if any part of the log device is at >> the other side of a >> WAN. So either add a mirror of log devices locally >> and not across the WAN, >> or don''t do it at all. >> >> >>> I am considering building a separate mirrored zpool >> of Flash disk that >>> span the frames, then creating zvols to use as log >> devices for the >>> data zpool. Will this work? Any other >> suggestions? >> >> This also sounds nonsensical to me. If your primary >> pool devices are Flash, >> then there''s no point to add separate log devices. >> Unless you have another >> ype of even faster nonvolatile storage. > > Both frames are FC connected with Flash devices in the frame. Latencies are additive, so there is benefit to a logging device. The cluster is a standard HA cluster about 10km apart with identical storage in both locations, mirrored using ZFS.There are quite a few metro clusters in the world today. Many use traditional mirroring software. Some use array-based sync replication. A ZFS-based solution works and behaves similarly.> Think about the potential problems if I don''t mirror the log devices across the WAN.If you use log devices, mirror them. -- richard -- Richard Elling richard at nexenta.com +1-760-896-4422 ZFS and NexentaStor training, Rotterdam, July 13-15, 2010 http://nexenta-rotterdam.eventbrite.com/
On Wed, May 19, 2010 at 07:50:13AM -0700, John Hoogerdijk wrote:> Think about the potential problems if I don''t mirror the log devices > across the WAN.If you don''t mirror the log devices then your disaster recovery semantics will be that you''ll miss any transactions that hadn''t been committed to disk yet at the time of the disaster. Which means that the log devices'' effects is purely local: for recovery from local power failures (not extending to local disasters) and for acceleration. This may or may not be acceptable to you. If not, then mirror the log devices. Nico --