This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Thanks. Sent from a fruity device
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Tiernan OToole > > This might be a stupid question, but here goes... Would adding, say, 4 4or> 8gb usb keys as a zil make enough of a difference for writes on an iscsishared> vol? > > I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives > stripped) but writes top out at about 10 and drop a lot lower... If Iwhere to> add a couple usb keys for zil, would it make a difference?Unfortunately, usb keys, even the fastest ones, are slower than physical hard drives. I even went out of my way to buy a super expensive super fast USB3 16G fob... And it''s still slower than a super-cheap USB2 sata hard drive. There is a way you can evaluate the effect of adding a fast slog device without buying one. (It would have to be a fast device, certainly no USB fobs.) Just temporarily disable your ZIL. That''s the fastest you can possibly go. If it makes a big difference, then getting a fast slog device will help you approach that theoretical limit. If it doesn''t make a huge difference, then adding slog will not do you any good. To disable ZIL, if your pool is sufficiently recent, use the zfs set synccommand. It takes effect immediately. If you have an older system, you''ll have to use a different command, and you''ll probably have to remount your filesystem in order for the change to take effect.
On Fri, Jul 15, 2011 at 04:21:13PM +0000, Tiernan OToole wrote:> This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? > > I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference?Speaking of which, is there a point in using an eSATA flash stick? If yes, which? -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
Well, not knowing a lot about these, but if the flash stick is based on SSD, then it might work well, but if its just a standard USB key rebundled as a eSATA disk, maybe not... On Fri, Jul 15, 2011 at 5:54 PM, Eugen Leitl <eugen at leitl.org> wrote:> On Fri, Jul 15, 2011 at 04:21:13PM +0000, Tiernan OToole wrote: > > This might be a stupid question, but here goes... Would adding, say, 4 4 > or 8gb usb keys as a zil make enough of a difference for writes on an iscsi > shared vol? > > > > I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives > stripped) but writes top out at about 10 and drop a lot lower... If I where > to add a couple usb keys for zil, would it make a difference? > > Speaking of which, is there a point in using an eSATA flash stick? > If yes, which? > > -- > Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org > ______________________________________________________________ > ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org > 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Tiernan O''Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110716/1994cfb9/attachment.html>
Thanks for the info. need to rebuild my machine and ZFS pool.... kind of new to this and realized i built it as a stripe, not a mirror... also, want to add extra disks... As a follow up question: I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would that work? is it even posible? or what would you recomend for that setup? Thanks. --Tiernan On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey < opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:> > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > > bounces at opensolaris.org] On Behalf Of Tiernan OToole > > > > This might be a stupid question, but here goes... Would adding, say, 4 4 > or > > 8gb usb keys as a zil make enough of a difference for writes on an iscsi > shared > > vol? > > > > I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives > > stripped) but writes top out at about 10 and drop a lot lower... If I > where to > > add a couple usb keys for zil, would it make a difference? > > Unfortunately, usb keys, even the fastest ones, are slower than physical > hard drives. I even went out of my way to buy a super expensive super fast > USB3 16G fob... And it''s still slower than a super-cheap USB2 sata hard > drive. > > There is a way you can evaluate the effect of adding a fast slog device > without buying one. (It would have to be a fast device, certainly no USB > fobs.) Just temporarily disable your ZIL. That''s the fastest you can > possibly go. If it makes a big difference, then getting a fast slog device > will help you approach that theoretical limit. If it doesn''t make a huge > difference, then adding slog will not do you any good. > > To disable ZIL, if your pool is sufficiently recent, use the zfs set sync> command. It takes effect immediately. If you have an older system, you''ll > have to use a different command, and you''ll probably have to remount your > filesystem in order for the change to take effect. > >-- Tiernan O''Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110716/7087dde5/attachment.html>
Craig Cory
2011-Jul-16 13:49 UTC
[zfs-discuss] (Zil on multiple usb keys) Mirroring the pool
Tiernan,
Depending on how you have created your current pool, you *may* be able to add
the mirroring without rebuilding it. Each disk in the stripe can have a second
disk of equal size attached to it to form a mirrored component, or vdev. So if
your pool has 2 500GB drives, attach another 500GB drive to each, forming a
mirror of each stripe half.
# zpool status
pool: mypool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
c0t50d1 ONLINE 0 0 0
c0t50d2 ONLINE 0 0 0
errors: No known data errors
# zpool attach mypool c0t50d1 c0t50d3
# zpool attach mypool c0t50d2 c0t50d4
# zpool status
pool: mypool
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Sat Jul 16 07:38:07 2011
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t50d1 ONLINE 0 0 0
c0t50d3 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t50d2 ONLINE 0 0 0
c0t50d4 ONLINE 0 0 0
errors: No known data errors
#
Both single vdevs (c0t50d1 and c0t50d2) are now mirrored.
If you don''t have a second disk appropriately sized to match the
current pool
members, you can create one or two pools with your two 500GB and two 300GB
disks,, depending on your needs.
Either:
# zpool create pool1 mirror <500GB-1> <500GB-2> mirror
<300GB-1> <300GB-2>
to make one ~800GB pool. Or
# zpool create pool1 mirror <500GB-1> <500GB-2>
# zpool create pool2 mirror <300GB-1> <300GB-2>
to make two pools, one ~500GB and one ~300GB.
As long as the mirrored pairs match they do not have to be all the same in the
pool.
Craig
Tiernan OToole wrote:> Thanks for the info. need to rebuild my machine and ZFS pool.... kind of
new
> to this and realized i built it as a stripe, not a mirror... also, want to
> add extra disks...
>
> As a follow up question:
>
> I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create
> a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would
> that work? is it even posible? or what would you recomend for that setup?
>
> Thanks.
>
> --Tiernan
>
> On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey <
> opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:
>
>> > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> > bounces at opensolaris.org] On Behalf Of Tiernan OToole
>> >
>> > This might be a stupid question, but here goes... Would adding,
say, 4 4
>> or
>> > 8gb usb keys as a zil make enough of a difference for writes on an
iscsi
>> shared
>> > vol?
>> >
>> > I am finding reads are not too bad (40is mb/s over gige on 2 500gb
drives
>> > stripped) but writes top out at about 10 and drop a lot lower...
If I
>> where to
>> > add a couple usb keys for zil, would it make a difference?
>>
>> Unfortunately, usb keys, even the fastest ones, are slower than
physical
>> hard drives. I even went out of my way to buy a super expensive super
fast
>> USB3 16G fob... And it''s still slower than a super-cheap USB2
sata hard
>> drive.
>>
>> There is a way you can evaluate the effect of adding a fast slog device
>> without buying one. (It would have to be a fast device, certainly no
USB
>> fobs.) Just temporarily disable your ZIL. That''s the fastest
you can
>> possibly go. If it makes a big difference, then getting a fast slog
device
>> will help you approach that theoretical limit. If it doesn''t
make a huge
>> difference, then adding slog will not do you any good.
>>
>> To disable ZIL, if your pool is sufficiently recent, use the zfs set
sync>> command. It takes effect immediately. If you have an older
system, you''ll
>> have to use a different command, and you''ll probably have to
remount your
>> filesystem in order for the change to take effect.
>>
>>
>
>
> --
> Tiernan O''Toole
> blog.lotas-smartman.net
> www.tiernanotoolephotography.com
> www.the-hairy-one.com
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
--
Craig Cory
Senior Instructor :: ExitCertified
: Oracle/Sun Certified System Administrator
: Oracle/Sun Certified Network Administrator
: Oracle/Sun Certified Security Administrator
: Symantec/Veritas Certified Instructor
: RedHat Certified Systems Administrator
8950 Cal Center Drive
Bldg 1, Suite 110
Sacramento, California 95826
[e] craig.cory at exitcertified.com
[p] 916.669.3970
[f] 916.669.3977
+-------------------------------------------------------------------------+
ExitCertified :: Excellence in IT Certified Education
Certified training with Oracle, Sun Microsystems, Apple, Symantec, IBM,
Red Hat, MySQL, Hitachi Storage, SpringSource and VMWare.
1.800.803.EXIT (3948) | www.ExitCertified.com
+-------------------------------------------------------------------------+
> From: Tiernan OToole [mailto:lsmartman at gmail.com] > Sent: Saturday, July 16, 2011 7:46 AM > > I have 2 500Gb internal drives and 2 300Gb USB drives. If i where tocreate a 2> pools, a 300Gb and a 500Gb in each, and then mirror over them, would that > work? is it even posible? or what would you recomend for that setup?I think the risk of accidental disconnection is higher on the USB drive. So I would recommend swapping the disks inside the enclosures... One 500 inside, one 500 outside, one 300 inside, one 300 outside. Mirror the 500G drives to each other, mirror the 300g drives to each other. That way, if you accidentally disconnect one or both of the external drives, you just plug it back in and everything moves forward without any problem.
2011-07-16 15:46, Tiernan OToole ?????:> Thanks for the info. need to rebuild my machine and ZFS pool.... kind > of new to this and realized i built it as a stripe, not a mirror... > also, want to add extra disks... > > As a follow up question: > > I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to > create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over > them, would that work? is it even posible? or what would you recomend > for that setup? >Is there a typo? It would rather be a 2*300Gb mirror and a 2*500Gb mirror, with a "stripe" above them as much as writes can get balanced. That would work (with forcing on command-line), is possible, moderately recommmended because unbalanced setups can have more issues than usual (hence you must use the force to enable such setup). And just in case, this pool can not be a bootable rpool. You might make a 2*200Gb slice mirror for an rpool and a more balanced 4*300Gb pool of any layout (raid10, raidz123)... As for using USB sticks, I started my unlucky setup with some sticks used as L2ARC, and about once a week the device got lost (possibly because a stick could slide a bit from its contact bay on the chassis - BIOS also did not see the stick until it was re-plugged). Loss of a device would also hang my pool for quite a long while...> Thanks. > > --Tiernan > > On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey > <opensolarisisdeadlongliveopensolaris at nedharvey.com > <mailto:opensolarisisdeadlongliveopensolaris at nedharvey.com>> wrote: > > > From: zfs-discuss-bounces at opensolaris.org > <mailto:zfs-discuss-bounces at opensolaris.org> [mailto:zfs-discuss- > <mailto:zfs-discuss-> > > bounces at opensolaris.org <mailto:bounces at opensolaris.org>] On > Behalf Of Tiernan OToole > > > > This might be a stupid question, but here goes... Would adding, > say, 4 4 > or > > 8gb usb keys as a zil make enough of a difference for writes on > an iscsi > shared > > vol? > > > > I am finding reads are not too bad (40is mb/s over gige on 2 > 500gb drives > > stripped) but writes top out at about 10 and drop a lot lower... > If I > where to > > add a couple usb keys for zil, would it make a difference? > > Unfortunately, usb keys, even the fastest ones, are slower than > physical > hard drives. I even went out of my way to buy a super expensive > super fast > USB3 16G fob... And it''s still slower than a super-cheap USB2 > sata hard > drive. > > There is a way you can evaluate the effect of adding a fast slog > device > without buying one. (It would have to be a fast device, certainly > no USB > fobs.) Just temporarily disable your ZIL. That''s the fastest you can > possibly go. If it makes a big difference, then getting a fast > slog device > will help you approach that theoretical limit. If it doesn''t make > a huge > difference, then adding slog will not do you any good. > > To disable ZIL, if your pool is sufficiently recent, use the zfs > set sync> command. It takes effect immediately. If you have an older > system, you''ll > have to use a different command, and you''ll probably have to > remount your > filesystem in order for the change to take effect. > > > > > -- > Tiernan O''Toole > blog.lotas-smartman.net <http://blog.lotas-smartman.net> > www.tiernanotoolephotography.com <http://www.tiernanotoolephotography.com> > www.the-hairy-one.com <http://www.the-hairy-one.com> > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- +============================================================+ | | | ?????? ???????, Jim Klimov | | ??????????? ???????? CTO | | ??? "??? ? ??" JSC COS&HT | | | | +7-903-7705859 (cellular) mailto:jimklimov at cos.ru | | CC:admin at cos.ru,jimklimov at mail.ru | +============================================================+ | () ascii ribbon campaign - against html mail | | /\ - against microsoft attachments | +============================================================+ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110716/f52495f3/attachment.html>
So, i like the sound of that, but the box is a very frankinbox like... it has 2 SATA ports, one used for the boot drive, one for one of the 500s... the second 500Gb is IDE. The 2 USB drives both internally are SATA, so pulling one and plugging it internally wont work that well... but thats for the info. On Sat, Jul 16, 2011 at 3:09 PM, Edward Ned Harvey < opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:> > From: Tiernan OToole [mailto:lsmartman at gmail.com] > > Sent: Saturday, July 16, 2011 7:46 AM > > > > I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to > create a 2 > > pools, a 300Gb and a 500Gb in each, and then mirror over them, would that > > work? is it even posible? or what would you recomend for that setup? > > I think the risk of accidental disconnection is higher on the USB drive. > So > I would recommend swapping the disks inside the enclosures... One 500 > inside, one 500 outside, one 300 inside, one 300 outside. Mirror the 500G > drives to each other, mirror the 300g drives to each other. That way, if > you accidentally disconnect one or both of the external drives, you just > plug it back in and everything moves forward without any problem. > >-- Tiernan O''Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110716/a9e70ceb/attachment-0001.html>
thats not a typo... I was thinking 2 pools, 800gb each, and mirrored... think i should mess around with this setup a bit more and see what i can get working... might work better if i just move them into a new enclosure... we see what happens... Thanks for the info on the USB drives... if the ZIL drive falls over, does ZFS not recover well? do i need to reboot fully? Thanks. --Tienan On Sat, Jul 16, 2011 at 3:23 PM, Jim Klimov <jimklimov at cos.ru> wrote:> 2011-07-16 15:46, Tiernan OToole ?????: > > Thanks for the info. need to rebuild my machine and ZFS pool.... kind of > new to this and realized i built it as a stripe, not a mirror... also, want > to add extra disks... > > As a follow up question: > > I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to > create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, > would that work? is it even posible? or what would you recomend for that > setup? > > Is there a typo? It would rather be a 2*300Gb mirror and a 2*500Gb > mirror, > with a "stripe" above them as much as writes can get balanced. > > That would work (with forcing on command-line), is possible, moderately > recommmended because unbalanced setups can have more issues than > usual (hence you must use the force to enable such setup). > > And just in case, this pool can not be a bootable rpool. > > You might make a 2*200Gb slice mirror for an rpool and a more balanced > 4*300Gb pool of any layout (raid10, raidz123)... > > As for using USB sticks, I started my unlucky setup with some sticks used > as L2ARC, and about once a week the device got lost (possibly because > a stick could slide a bit from its contact bay on the chassis - BIOS also > did > not see the stick until it was re-plugged). Loss of a device would also > hang > my pool for quite a long while... > > Thanks. > > --Tiernan > > On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey < > opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote: > >> > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> > bounces at opensolaris.org] On Behalf Of Tiernan OToole >> > >> > This might be a stupid question, but here goes... Would adding, say, 4 4 >> or >> > 8gb usb keys as a zil make enough of a difference for writes on an iscsi >> shared >> > vol? >> > >> > I am finding reads are not too bad (40is mb/s over gige on 2 500gb >> drives >> > stripped) but writes top out at about 10 and drop a lot lower... If I >> where to >> > add a couple usb keys for zil, would it make a difference? >> >> Unfortunately, usb keys, even the fastest ones, are slower than physical >> hard drives. I even went out of my way to buy a super expensive super >> fast >> USB3 16G fob... And it''s still slower than a super-cheap USB2 sata hard >> drive. >> >> There is a way you can evaluate the effect of adding a fast slog device >> without buying one. (It would have to be a fast device, certainly no USB >> fobs.) Just temporarily disable your ZIL. That''s the fastest you can >> possibly go. If it makes a big difference, then getting a fast slog >> device >> will help you approach that theoretical limit. If it doesn''t make a huge >> difference, then adding slog will not do you any good. >> >> To disable ZIL, if your pool is sufficiently recent, use the zfs set sync>> command. It takes effect immediately. If you have an older system, >> you''ll >> have to use a different command, and you''ll probably have to remount your >> filesystem in order for the change to take effect. >> >> > > > -- > Tiernan O''Toole > blog.lotas-smartman.net > www.tiernanotoolephotography.com > www.the-hairy-one.com > > > _______________________________________________ > zfs-discuss mailing listzfs-discuss at opensolaris.orghttp://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > -- > > > +============================================================+ > | | > | ?????? ???????, Jim Klimov | > | ??????????? ???????? CTO | > | ??? "??? ? ??" JSC COS&HT | > | | > | +7-903-7705859 (cellular) mailto:jimklimov at cos.ru <jimklimov at cos.ru> | > | CC:admin at cos.ru,jimklimov at mail.ru | > +============================================================+ > | () ascii ribbon campaign - against html mail | > | /\ - against microsoft attachments | > +============================================================+ > > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >-- Tiernan O''Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110716/a2416dec/attachment.html>
2011-07-16 18:48, Tiernan OToole ?????:> thats not a typo... I was thinking 2 pools, 800gb each, and > mirrored... think i should mess around with this setup a bit more and > see what i can get working... might work better if i just move them > into a new enclosure... we see what happens... >Well, in terms of mirroring over stripes, if any component of any stripe breaks, the whole half of the mirror is degraded. If another drive from another half also breaks, you''re in trouble. Overall raid01 is assumed to be less reliable than raid10, especially if there are more than two couples of drives.> Thanks for the info on the USB drives... if the ZIL drive falls over, > does ZFS not recover well? do i need to reboot fully? >In theory, you should have 2 drives (slices) mirrored for a ZIL. If one breaks, the other goes on. If all break, the pool reverts to on-disk ZIL areas in the course of several TXG syncs (a few seconds). So the big problem is constrained in poweroff/panic of the system in these few seconds when no ZIL is used. But in practice it may take ZFS a long time to detect the failure/death of the device, because it is not ZFS itself doing it - lower-level drivers like "sd" must detect and report the problem or the timeout... After all, I did not try a ZIL, or not for a long enough time to be certain of anything. From theory again, its critical property is fast writes with low latency, and reliability. Even in comparison with a dedicated HDD slice, my USB stick could not compete with either. But that was only a test setup, so the sticks were not "optimized" or "fast" (3-7Mb/s). You might want to look at CF-IDE adapters - new professional photo CF cards mention 600x (90Mb/s) speeds at reads and 30 to 90Mb/s writes, and some internal redundancy. Probably not cheap though... But many small mobos include a CF port, or a small adapter can be plugged into an IDE port. There were some adapters which naturally "split" one IDE port into master and slave CF ports. Or see a PCMCIA-CF port for a laptop... Or maybe even use a USB-CF card reader thingie. -- +============================================================+ | | | ?????? ???????, Jim Klimov | | ??????????? ???????? CTO | | ??? "??? ? ??" JSC COS&HT | | | | +7-903-7705859 (cellular) mailto:jimklimov at cos.ru | | CC:admin at cos.ru,jimklimov at mail.ru | +============================================================+ | () ascii ribbon campaign - against html mail | | /\ - against microsoft attachments | +============================================================+
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Jim Klimov > > Well, in terms of mirroring over stripes, if any component of any > stripe > breaks, > the whole half of the mirror is degraded. If another drive from another > half > also breaks, you''re in trouble.There''s no such thing in ZFS. You have a series of mirrors in the pool, and if one side of one mirror breaks, no big deal. If another side of another mirror breaks, no big deal. The only problem is when you lose both sides of a single mirror. Now please, nobody say this is either raid10 or raid01. Because it''s neither one. The definition of striping according to raid0 does not exist in ZFS - but the essence is preserved and improved upon. In a strictly defined raid0, you have a set number of devices which are all the same size. It benefits large sequential operations, but it hurts small operations. You cannot expand by simply adding more devices, and they all must be the same size. The ZFS concept of striping is more like a combination of raid0 striping and concatenation ... Preserve the best parts of each and throw away the bad parts of each. Optimize for both small operations and serial operations, expand with any size disk at any time.
2011-07-16 20:42, Edward Ned Harvey ?????:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Jim Klimov >> >> Well, in terms of mirroring over stripes, if any component of any >> stripe >> breaks, >> the whole half of the mirror is degraded. If another drive from another >> half >> also breaks, you''re in trouble. > There''s no such thing in ZFS. You have a series of mirrors in the pool, and > if one side of one mirror breaks, no big deal. If another side of another > mirror breaks, no big deal. The only problem is when you lose both sides of > a single mirror. > > Now please, nobody say this is either raid10 or raid01. Because it''s > neither one.Well, I do stand corrected in case of a standard ZFS layout which would automatically make the zfs-striping over mirrors of identical disks and not vice-versa. As a lame defense for myself, I can say that the dangers described apply to the case that, if the OP were so inclined, he could craft a couple of "striped" pools (300+500) and then make a ZFS pool over these two. But yes, it would take some effort and determination to make such an ugly construct in the first place ;) //Jim
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Jim Klimov > > if the OP were so inclined, > he could craft a couple of "striped" pools (300+500) and > then make a ZFS pool over these two.Actually, you can''t do that. You can''t make a vdev from other vdev''s, and when it comes to striping and mirroring your only choice is to do it the right way. If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol''s and no iscsi... But either way you''d really be going out of your way to create a problem. ;-)
2011-07-17 23:13, Edward Ned Harvey ?????:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Jim Klimov >> >> if the OP were so inclined, >> he could craft a couple of "striped" pools (300+500) and >> then make a ZFS pool over these two. > Actually, you can''t do that. You can''t make a vdev from other vdev''s, and when it comes to striping and mirroring your only choice is to do it the right way.Yup, that''s what i said later in the post.> > If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol''s and no iscsi... But either way you''d really be going out of your way to create a problem. ;-) >Theoretically, at least, you could do that. AFAIK you can even do it without iSCSI by just addressing zvol device names with their full paths, an yes after crafting yourself an intricate problem you''d likely get one ;) Not that it''s impossible or forbidden to shoot oneself in the foot, as your and my adventures demonstrate quite well ;) //Jim
On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:> Actually, you can''t do that. ?You can''t make a vdev from other vdev''s, and when it comes to striping and mirroring your only choice is to do it the right way. > > If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ? ;-) ?You might even be able to do the same crazy thing with simply zvol''s and no iscsi... ?But either way you''d really be going out of your way to create a problem. ? ;-)The right way to do it, um, incorrectly is to create a striped device using SVM, and use that as a vdev for your pool. So yes, you could create two 800GB stripes, and use them to create a ZFS mirror. But it would be a really bad idea. -B -- Brandon High : bhigh at freaks.com
On Mon, Jul 18, 2011 at 1:20 PM, Brandon High <bhigh at freaks.com> wrote:> On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey > <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote: >> Actually, you can''t do that. ?You can''t make a vdev from other vdev''s, and when it comes to striping and mirroring your only choice is to do it the right way. >> >> If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ? ;-) ?You might even be able to do the same crazy thing with simply zvol''s and no iscsi... ?But either way you''d really be going out of your way to create a problem. ? ;-) > > The right way to do it, um, incorrectly is to create a striped device > using SVM, and use that as a vdev for your pool. > > So yes, you could create two 800GB stripes, and use them to create a > ZFS mirror. But it would be a really bad idea. >In freebsd you can use geom_stripe or geom_concat to create striped block device.
Ok, so, taking 2 300Gb disks, and 2 500Gb disks, and creating an 800Gb mirrored striped thing is sounding like a bad idea... what about just creating a pool of all disks, without using mirrors? I seen something called "copies", which if i am reading correctly, will make sure a number of copies of a file exist... Am i reading that correctly? If this does work the way i think it works, then taking all 4 disks, and making one large 1.6Tb pool, setting copies to 2, should, in theory, create a poor mans pool with striping, right? --Tiernan On Mon, Jul 18, 2011 at 7:20 AM, Brandon High <bhigh at freaks.com> wrote:> On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey > <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote: > > Actually, you can''t do that. You can''t make a vdev from other vdev''s, > and when it comes to striping and mirroring your only choice is to do it the > right way. > > > > If you were REALLY trying to go out of your way to do it wrong somehow, I > suppose you could probably make a zvol from a stripe, and then export it to > yourself via iscsi, repeat with another zvol, and then mirror the two iscsi > targets. ;-) You might even be able to do the same crazy thing with > simply zvol''s and no iscsi... But either way you''d really be going out of > your way to create a problem. ;-) > > The right way to do it, um, incorrectly is to create a striped device > using SVM, and use that as a vdev for your pool. > > So yes, you could create two 800GB stripes, and use them to create a > ZFS mirror. But it would be a really bad idea. > > -B > > -- > Brandon High : bhigh at freaks.com > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Tiernan O''Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110718/9474f2d3/attachment.html>
On Mon, Jul 18, 2011 at 3:28 PM, Tiernan OToole <lsmartman at gmail.com> wrote:> Ok, so, taking 2 300Gb disks, and 2 500Gb disks, and creating an 800Gb > mirrored striped thing is sounding like a bad idea... what about just > creating a pool of all disks, without using mirrors? I seen something called > "copies", which if i am reading correctly, will make sure a number of copies > of a file exist... Am i reading that correctly? If this does work the way i > think it works, then taking all 4 disks, and making one large 1.6Tb pool, > setting copies to 2, should, in theory, create a poor mans pool with > striping, right?Step back a moment. What are your priorites? What is the most important thing for you? Is it space? Is it data protection? Is it something else? Once you determine that, it''s easier to come up with a reasonable setup. Back to your original question, I''d like to note some things. First of all, using USB disks for permanent storage is a bad idea. Go for e-sata instead (http://en.wikipedia.org/wiki/Serial_ata#eSATA). It eliminates the overhead caused by USB-to-[P/S]ATA bridge. You can get something like a e-sata bracket (If your controller supports port multiplier) or e-sata PCI controller, plus e-sata enclosure with 2 or 4 drive bays (depending on your needs). Second, using copies=2 + stripe is, again, a bad idea. While "copies=2" can protect you from something like bad sector, it will NOT protect you from drive failure. So when one drive broke your pool will still be unaccessible. Stick with strpe of mirrors instead. Go with what Edward suggested: rearrange the disk, and create stripe of (mirror of 500G internal + 500G external) + (mirror of 300G internal + 300G external). Another option, if you go with external enclosure route, is to just put all disks in the external enclosure, and go with the configuration Jim suggested (2 x 200GB mirror, plus 4 x 300GB mirror/raidz1) -- Fajar
2011/7/15 Eugen Leitl <eugen at leitl.org>:> Speaking of which, is there a point in using an eSATA flash stick? > If yes, which?It depends on the drive off course, you''ll have to look up benchmark results - but there are eSata sticks out there that are more or less built to Perform (as opposed to providing cheap storage). -- Frank Van Damme No part of this copyright message may be reproduced, read or seen, dead or alive or by any means, including but not limited to telepathy without the benevolence of the author.
If using two mirrors, you''ll end up with a badly balanced pool. As it was, and possibly is, this will lead to a performance penalty when one VDEV is full (the 300GB VDEV). This write performance has reportedly been fixed in Illumos, but I don''t know about S11ex. For OpenIndiana/Nexenta, the fix probably isn''t in yet. roy ----- Original Message -----> Ok, so, taking 2 300Gb disks, and 2 500Gb disks, and creating an 800Gb > mirrored striped thing is sounding like a bad idea... what about just > creating a pool of all disks, without using mirrors? I seen something > called "copies", which if i am reading correctly, will make sure a > number of copies of a file exist... Am i reading that correctly? If > this does work the way i think it works, then taking all 4 disks, and > making one large 1.6Tb pool, setting copies to 2, should, in theory, > create a poor mans pool with striping, right? > --Tiernan > On Mon, Jul 18, 2011 at 7:20 AM, Brandon High < bhigh at freaks.com > > wrote: > > On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey > > < opensolarisisdeadlongliveopensolaris at nedharvey.com > wrote: > > > Actually, you can''t do that. You can''t make a vdev from other > > > vdev''s, and when it comes to striping and mirroring your only > > > choice > > > is to do it the right way. > > > > > > If you were REALLY trying to go out of your way to do it wrong > > > somehow, I suppose you could probably make a zvol from a stripe, > > > and > > > then export it to yourself via iscsi, repeat with another zvol, > > > and > > > then mirror the two iscsi targets. ;-) You might even be able to > > > do > > > the same crazy thing with simply zvol''s and no iscsi... But either > > > way you''d really be going out of your way to create a problem. ;-) > > The right way to do it, um, incorrectly is to create a striped > > device > > using SVM, and use that as a vdev for your pool. > > So yes, you could create two 800GB stripes, and use them to create a > > ZFS mirror. But it would be a really bad idea. > > -B > > -- > > Brandon High : bhigh at freaks.com > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > -- > Tiernan O''Toole > blog.lotas-smartman.net > www.tiernanotoolephotography.com > www.the-hairy-one.com > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110718/a7dfd4d0/attachment-0001.html>
+1 on the below, and in addition... ...compact flash, like off of USB sticks is not designed to deal with very many writes to it. Commonly it is used to store a bootable image that maybe once a year will have an upgrade on it. Basically, trying to use those devices for a ZIL, even they are mirrored - you should be prepared to having one die and be replaced very, very regularly. Generally performance is going to pretty bad as well - USB sticks are not made to be written too rapidly. They are entirely different animals than SSDs. I would not be surprised (but would be curious to know if you still move forward on this) that you will find performance even worse trying to do this. On Jul 18, 2011, at 1:54 AM, Fajar A. Nugraha wrote:> First of all, using USB disks for permanent storage is a bad idea. Go > for e-sata instead (http://en.wikipedia.org/wiki/Serial_ata#eSATA). It-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110722/5ece57b2/attachment.html>
On 07/23/11 04:57, Michael DeMan wrote:> Generally performance is going to pretty bad as well - USB sticks are > not made to be written too rapidly. They are entirely different animals > than SSDs. I would not be surprised (but would be curious to know if you > still move forward on this) that you will find performance even worse > trying to do this.Back in the snv_120 ish era I tried this experiement on both my pool and on a friends. In both cases we were serving NFS (he was also doing CIFS) which was mostly read but also had periods where 1-2 G of data was rapidly added (uploading photos or videos) over the network. In both the USB "flash drive" and in the case of a San Disk Extreme IV CF card in a CF->IDE enclosure the performance did not improve and in fact in the case of the CF card the enclosure was bugging such that the changes we had to make to the ata config did actually make it slower. I removed the separate log device from both of those pools (by manual hacking with specially build zfs kernel modules because slog removal didn''t exist back then.). -- Darren J Moffat