Hi, I know the general discussion is about flash SSD''s connected through SATA/SAS or possibly PCI-E these days. So excuse me if I''m askign something that makes no sense... I have a server that can hold 6 U320 SCSI disks. Right now I put in 5 300GB for a data pool, and 1 18GB for the root pool. I''ve been thinking lately that I''m not sure I like the root pool being unprotected, but I can''t afford to give up another drive bay. So recently the idea occurred to me to go the other way. If I were to get 2 USB Flash Thunb drives say 16 or 32 GB each, not only would i be able to mirror the root pool, but I''d also be able to put a 6th 300GB drive into the data pool. That led me to wonder whether partitioning out 8 or 12 GB on a 32GB thumb drive would be beneficial as an slog?? I bet the USB bus won''t be as good as SATA or SAS, but will it be better than the internal ZIL on the U320 drives? This seems like at least a "win-win", and possibly a "win-win-win". Is there some other reason I''m insane to consider this? -Kyle
The last couple times i''ve read this questions, people normally responded with: It depends.... you might not even NEED a slog, there is a script floating around which can help determine that... If you could benefit from one, it''s going to be IOPS which help you....so if the usb drive has more iops than your pool configuration does, then it might give some benefit.....but then again, usb might not be as safe either, and if an older version you may want to mirror it. On Tue, May 25, 2010 at 8:11 AM, Kyle McDonald <kmcdonald at egenera.com>wrote:> Hi, > > I know the general discussion is about flash SSD''s connected through > SATA/SAS or possibly PCI-E these days. So excuse me if I''m askign > something that makes no sense... > > I have a server that can hold 6 U320 SCSI disks. Right now I put in 5 > 300GB for a data pool, and 1 18GB for the root pool. > > I''ve been thinking lately that I''m not sure I like the root pool being > unprotected, but I can''t afford to give up another drive bay. So > recently the idea occurred to me to go the other way. If I were to get 2 > USB Flash Thunb drives say 16 or 32 GB each, not only would i be able to > mirror the root pool, but I''d also be able to put a 6th 300GB drive into > the data pool. > > That led me to wonder whether partitioning out 8 or 12 GB on a 32GB > thumb drive would be beneficial as an slog?? I bet the USB bus won''t be > as good as SATA or SAS, but will it be better than the internal ZIL on > the U320 drives? > > This seems like at least a "win-win", and possibly a "win-win-win". > Is there some other reason I''m insane to consider this? > > -Kyle > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100525/85038469/attachment.html>
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Kyle McDonald > > I''ve been thinking lately that I''m not sure I like the root pool being > unprotected, but I can''t afford to give up another drive bay.I''m guessing you won''t be able to use the USB thumbs as a boot device. But that''s just a guess. However, I see nothing wrong with mirroring your primary boot device to the USB. At least in this case, if the OS drive fails, your system doesn''t crash. You''re able to swap the OS drive and restore your OS mirror.> That led me to wonder whether partitioning out 8 or 12 GB on a 32GB > thumb drive would be beneficial as an slog??I think the only way to find out is to measure it. I do have an educated guess though. I don''t think, even the fastest USB flash drives are able to work quickly, with significantly low latency. Based on measurements I made years ago, so again I emphasize, only way to find out is to test it. One thing you could check, which does get you a lot of mileage for "free" is: Make sure your HBA has a BBU, and enable the WriteBack. In my measurements, this gains about 75% of the benefit that log devices would give you.
The USB stack in OpenSolaris is ... complex (STREAMs based!), and
probably not the most performant or reliable portion of the system.
Furthermore, the mass storage layer, which encapsulates SCSI, is not
tuned for a high number of IOPS or low latencies, and the stack makes
different assumptions about USB media than it makes for SCSI. Further,
you will not be able to get direct DMA through this stack either, so you
wind up sucking extra CPU doing data copies. I would think long and hard
before I put too many eggs in that particular basket.
Additionally, USB has the tendency to run at high interrupt rates (1000
Hz), which can have a detrimental impact on system performance and power
consumption. Its possible that mass storage devices don''t have this
attribute -- I''m not sure, I''ve not tried to investigate it
directly.
One attribute that you can rest assured of though, is that the average
latency for USB operations cannot be less than 1 ms -- which is driven
by that 1000 Hz, because USB doesn''t have a "true" interrupt
mechanism
(it polls). I believe that this is considerably higher than the lowest
latency achievable with PCI and SATA or SAS devices.
Generally, eSATA flash drives would be preferable for external flash
media, I think. Additionally, the SATA framework has quite recently
inherited FMA support, so you''ll benefit from closer integration of FMA
and ZFS when using SATA.
- Garrett
On 5/25/2010 8:39 AM, Edward Ned Harvey wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Kyle McDonald
>>
>> I''ve been thinking lately that I''m not sure I like
the root pool being
>> unprotected, but I can''t afford to give up another drive bay.
>>
> I''m guessing you won''t be able to use the USB thumbs as a
boot device. But
> that''s just a guess.
>
> However, I see nothing wrong with mirroring your primary boot device to the
> USB. At least in this case, if the OS drive fails, your system
doesn''t
> crash. You''re able to swap the OS drive and restore your OS
mirror.
>
>
>
>> That led me to wonder whether partitioning out 8 or 12 GB on a 32GB
>> thumb drive would be beneficial as an slog??
>>
> I think the only way to find out is to measure it. I do have an educated
> guess though. I don''t think, even the fastest USB flash drives
are able to
> work quickly, with significantly low latency. Based on measurements I made
> years ago, so again I emphasize, only way to find out is to test it.
>
> One thing you could check, which does get you a lot of mileage for
"free"
> is: Make sure your HBA has a BBU, and enable the WriteBack. In my
> measurements, this gains about 75% of the benefit that log devices would
> give you.
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>
On 5/25/2010 11:39 AM, Edward Ned Harvey wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Kyle McDonald >> >> I''ve been thinking lately that I''m not sure I like the root pool being >> unprotected, but I can''t afford to give up another drive bay. >> > I''m guessing you won''t be able to use the USB thumbs as a boot device. But > that''s just a guess. >No I''ve installed to an 8GB one on my laptop and booted from it. And this server offers USB drives as a boot option, I don''t see why it wouldn''t work. but I won''t kow till I try it.> However, I see nothing wrong with mirroring your primary boot device to the > USB. At least in this case, if the OS drive fails, your system doesn''t > crash. You''re able to swap the OS drive and restore your OS mirror. > >True. If nothing else I may do at least that.> >> That led me to wonder whether partitioning out 8 or 12 GB on a 32GB >> thumb drive would be beneficial as an slog?? >> > I think the only way to find out is to measure it. I do have an educated > guess though. I don''t think, even the fastest USB flash drives are able to > work quickly, with significantly low latency. Based on measurements I made > years ago, so again I emphasize, only way to find out is to test it. > >Yes I guess I"ll have to try some benchmarks. The thing that got me thinking was that many of these drives support a windows feature called ''Ready boost'' - which I think is just windows swapping to the USB drive instead of HD - but Windows does a performance test on the device to seee it''s fast enough. I thought maybe if it''s faster to swap to than a HD it might be faster for an SLOG too. But you''re right the only way to know is to measure it.> One thing you could check, which does get you a lot of mileage for "free" > is: Make sure your HBA has a BBU, and enable the WriteBack. In my > measurements, this gains about 75% of the benefit that log devices would > give you. > >My HBA''s have 256MB of BBC. And it''s enabled on all 6 drives, so that should help. However I may have hit a bug inthe ''isp'' driver (still have to debug and see if that''s the root cause) and I may need to yank the RAID enabler, and go back to straight SCSI. -Kyle
Edward Ned Harvey wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Kyle McDonald >> >> I''ve been thinking lately that I''m not sure I like the root pool being >> unprotected, but I can''t afford to give up another drive bay. >> > > I''m guessing you won''t be able to use the USB thumbs as a boot device. But > that''s just a guess. > > However, I see nothing wrong with mirroring your primary boot device to the > USB. At least in this case, if the OS drive fails, your system doesn''t > crash. You''re able to swap the OS drive and restore your OS mirror. > > > >> That led me to wonder whether partitioning out 8 or 12 GB on a 32GB >> thumb drive would be beneficial as an slog?? >> > > I think the only way to find out is to measure it. I do have an educated > guess though. I don''t think, even the fastest USB flash drives are able to > work quickly, with significantly low latency. Based on measurements I made > years ago, so again I emphasize, only way to find out is to test it. > > One thing you could check, which does get you a lot of mileage for "free" > is: Make sure your HBA has a BBU, and enable the WriteBack. In my > measurements, this gains about 75% of the benefit that log devices would > give you. >There are or at least have been some issues with ZFS and devices. Here''s one that is still open: Bug 4755 - ZFS boot does not work with removable media (usb flash memory) http://defect.opensolaris.org/bz/show_bug.cgi?id=4755 Regarding performance...USB flash drives vary significantly in performance from one another between brands and models. Some get close to USB 2.0 theoretical limits, others just barely exceed USB 1.1. Vista and Windows 7 support the use of USB flash drives for ReadyBoost, a caching system to reduce application load times. Windows tests have shown that with enough RAM, that ReadyBoost caching offers little additional performance (as Windows does make use of system RAM for file caching too). I think using good USB flash drives has the potential to improve performance, and if you can keep mirrored flash drives on different, dedicated USB controllers that will help performance the most. If USB support in OpenSolaris has is poor and has weak performance, I wonder if an iSCSI target created out of the USB device on a Linux or Windows system on the same network might be able to offer better performance. Even if latency goes to 2-3ms, that''s still much better than the 8.5 ms random seek times on a 7200 rpm hard disk. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100525/3ca532d0/attachment.html>