I''m putting together a NexentaOS (b65)-based server that has 4 500 GB drives on it. Currently it has two, set up as a ZFS mirror. I''m able to boot Nexenta from it, and it seems to work ok. But, as I''ve learned, the mirror is not properly redundant, and so I can''t just have a drive fail (when I pull one, the OS ends up hanging, and even if I replace it, I have to reboot. I asked about this on the Nexenta list: http://www.gnusolaris.org/phpbb/viewtopic.php?t=6233). I think the problems stem from limitations in booting from ZFS. So, I''m going to re-do the drive allocations, but I''m not sure the best way to go. I''m hoping this list will give me some advice. I was thinking I''d put the basic OS on one drive (possibly a ZFS pool), and then make a RAID-Z storage pool from the other three drives. The RAID-Z pool would contain filesystems for /usr, /home, etc. The drawback to this approach is that a 500 GB drive is far too large for the basic OS, and seems to end up wasted. An alternative would be to partition one drive, give it 20 GB for the OS and the rest for use by ZFS to allocated to a storage pool, but at this point ZFS isn''t working with the raw device, and so I''m not sure what limitations this may place on me. BTW, I don''t mind if the boot drive fails, because it will be fairly easy to replace, and this server is only mission-critical to me and my friends. So...suggestions? What''s a good way to utilize the power and glory of ZFS in a 4x 500 GB system, without unnecessary waste? Thanks much! This message posted from opensolaris.org
Hi Rick, what do you think about this configuration: Part all disks like this 7GiB 493GiB Make a RAIDz1 out of the 493GiB partitions and a RAID5 out of the 7GiB partitions. Create a swap, and the root in the RAID5, the dirs with the user data in the ZFS storage. Backup the / daily to the ZFS :) Greetings Cyron> I''m putting together a NexentaOS (b65)-based server that has 4 500 GB > drives on it. Currently it has two, set up as a ZFS mirror. I''m able > to boot Nexenta from it, and it seems to work ok. But, as I''ve > learned, the mirror is not properly redundant, and so I can''t just > have a drive fail (when I pull one, the OS ends up hanging, and even > if I replace it, I have to reboot. I asked about this on the Nexenta > list: http://www.gnusolaris.org/phpbb/viewtopic.php?t=6233). > > I think the problems stem from limitations in booting from ZFS. So, > I''m going to re-do the drive allocations, but I''m not sure the best > way to go. I''m hoping this list will give me some advice. > > I was thinking I''d put the basic OS on one drive (possibly a ZFS > pool), and then make a RAID-Z storage pool from the other three > drives. The RAID-Z pool would contain filesystems for /usr, /home, > etc. The drawback to this approach is that a 500 GB drive is far too > large for the basic OS, and seems to end up wasted. > > An alternative would be to partition one drive, give it 20 GB for the > OS and the rest for use by ZFS to allocated to a storage pool, but at > this point ZFS isn''t working with the raw device, and so I''m not sure > what limitations this may place on me. > > BTW, I don''t mind if the boot drive fails, because it will be fairly > easy to replace, and this server is only mission-critical to me and > my friends. > > So...suggestions? What''s a good way to utilize the power and glory of > ZFS in a 4x 500 GB system, without unnecessary waste? > > Thanks much! > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss--- fortune --- Alle Menschen, die sonst nicht verhindern, wollen nun Aids vernindern. -- Werner Schneider -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070615/a7c80612/attachment.bin>
> Hi Rick, > > what do you think about this configuration: > > Part all disks like this > 7GiB > 493GiB > > Make a RAIDz1 out of the 493GiB partitions and a > RAID5 out of the 7GiB > partitions. Create a swap, and the root in the RAID5, > the dirs with the > user data in the ZFS storage. > > Backup the / daily to the ZFS :)Hmm. Not sure I can do RAID5 (and boot from it). Presumably, though, this would continue to function if a drive went bad. It also prevents ZFS from managing the devices itself, which I think is undesirable (according to the ZFS Admin Guide). I''m also not sure if I have RAID5 support in the BIOS. I think it''s just RAID0/1. Oh, how I long for full boot support in ZFS... This message posted from opensolaris.org
Rick Mann wrote:> > BTW, I don''t mind if the boot drive fails, because it will be fairly easy to replace, and this server is only mission-critical to me and my friends. > > So...suggestions? What''s a good way to utilize the power and glory of ZFS in a 4x 500 GB system, without unnecessary waste? > >Bung in (add a USB one if you don''t have space) a small boot drive and use all the others for for ZFS. Ian
Ian Collins wrote:> Bung in (add a USB one if you don''t have space) a small boot drive and > use all the others for for ZFS.Not a bad idea; I''ll have to see where I can put one. But, I thought I read somewhere that one can''t use ZFS for swap. Or maybe I read this: "Slices should only be used under the following conditions: * The device name is nonstandard. * A single disk is shared between ZFS and another file system, such as UFS. * A disk is used as a swap or a dump device." I definitely [i]don''t[/i] want to use flash for swap... This message posted from opensolaris.org
> Rick Mann wrote: > > > > BTW, I don''t mind if the boot drive fails, because it will be fairly easy to replace, and this server is only mission-critical to me and my friends. > > > > So...suggestions? What''s a good way to utilize the power and glory of ZFS in a 4x 500 GB system, without unnecessary waste? > > > Bung in (add a USB one if you don''t have space) a small boot drive and > use all the others for for ZFS.Having the ability to do something like a pivot root would make that even more useful. -- Darren Dunham ddunham at taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
Ian Collins wrote:> Rick Mann wrote: >> BTW, I don''t mind if the boot drive fails, because it will be fairly easy to replace, and this server is only mission-critical to me and my friends. >> >> So...suggestions? What''s a good way to utilize the power and glory of ZFS in a 4x 500 GB system, without unnecessary waste? >> >> > Bung in (add a USB one if you don''t have space) a small boot drive and > use all the others for for ZFS. > > Ian >This is how I run my home server w/ 4 500GB drives - a small 40GB IDE drive provides root & swap/dump device, the 4 500 GB drives are RAIDZ & contain all the data. I ran out of drive bays, so I used one of those 5 1/4" -> 3.5" adaptor brackets to hang the boot drive where a second DVD drive would go... - Bart -- Bart Smaalders Solaris Kernel Performance barts at cyber.eng.sun.com http://blogs.sun.com/barts
On Thu, 2007-06-14 at 17:45 -0700, Bart Smaalders wrote:> This is how I run my home server w/ 4 500GB drives - a small > 40GB IDE drive provides root & swap/dump device, the 4 500 GB > drives are RAIDZ & contain all the data. I ran out of drive > bays, so I used one of those 5 1/4" -> 3.5" adaptor brackets > to hang the boot drive where a second DVD drive would go...I''m doing something very similar (I actually used your blog post as the start of my shopping list). I''d like to replace the boot drive with a compact flash card but run with zfs root once booted, but haven''t had the time to investigate whether there''s enough left over from the initial zfs root integration to permit the ufs boot/zfs root hack (or some other "pivot" hack) to work.. If all you need to put on the CF is grub, a "unix" binary, and the boot archive, I should be able to use a relatively small card and still have enough room for a couple copies of everything. - Bill
Rick Mann wrote:> Ian Collins wrote: > > >> Bung in (add a USB one if you don''t have space) a small boot drive and >> use all the others for for ZFS. >> > > Not a bad idea; I''ll have to see where I can put one. > > But, I thought I read somewhere that one can''t use ZFS for swap. Or maybe I read this: > >I wouldn''t bother, just spec the machine with enough RAM so swap''s only real use is as a dump device. You can always use a swap file if you have to. Ian
Ian Collins wrote:> Rick Mann wrote: >> Ian Collins wrote: >> >> >>> Bung in (add a USB one if you don''t have space) a small boot drive and >>> use all the others for for ZFS. >>> >> Not a bad idea; I''ll have to see where I can put one. >> >> But, I thought I read somewhere that one can''t use ZFS for swap. Or maybe I read this: >> >> > I wouldn''t bother, just spec the machine with enough RAM so swap''s only > real use is as a dump device. You can always use a swap file if you > have to. > > IanIf you compile stuff (like opensolaris), you''ll want swap space. Esp. if you use dmake; 30 parallel C++ compilations can use up a lot of RAM. - Bart -- Bart Smaalders Solaris Kernel Performance barts at cyber.eng.sun.com http://blogs.sun.com/barts
Bart Smaalders wrote:> Ian Collins wrote: >> Rick Mann wrote: >>> Ian Collins wrote: >>> >>> >>>> Bung in (add a USB one if you don''t have space) a small boot drive and >>>> use all the others for for ZFS. >>>> >>> Not a bad idea; I''ll have to see where I can put one. >>> >>> But, I thought I read somewhere that one can''t use ZFS for swap. Or >>> maybe I read this: >>> >>> >> I wouldn''t bother, just spec the machine with enough RAM so swap''s only >> real use is as a dump device. You can always use a swap file if you >> have to. >> >> Ian > > If you compile stuff (like opensolaris), you''ll want swap space. > Esp. if you use dmake; 30 parallel C++ compilations can use up a > lot of RAM. >I should have said "I wouldn''t bother with swap on ZFS". If your compiles use swap, you are either doing too many jobs, or you don''t have enough RAM! Saying that, I did have one C++ file that used swap on my laptop with 1G of RAM and that was on a serial make. Ian
As I understand matters, from my notes to design the "perfect" home
NAS
server :-)
1) you want to give ZFS entire spindles if at all possible; that will
mean it can enable and utilise the drive''s hardware write cache
properly, leading to a performance boost. You want to do this if you
can. Alas it knocks out the "split all disks into 7 & 493Gb
partitions" design concept.
2) I''ve considered pivot-root solutions based around a USB stick or
drive; cute, but I want a single tower box and no "dongles"
3) This leads me to the following design points:
- enormous tower case with 10+ bays
- HE/high-efficency mobo with 8+ SATA capability
- crank down the CPU, big fans, etc... quiet....
- 1x [small/cheap]Gb Drive @ 10000+rpm for root / swap / alternate
boot environments
- 4x 750Gb SATA @ 7200rpm for full-spindle RAID-Z
- populate the spare SATA ports when 1Tb disks hit the price point;
make a separate RAIDZ and drop *that* into the existing pool.
This - curiously - echoes the Unixes of my youth (and earlier!) where
"root" was a small fast disk for swapping and access to key utilities
which were used frequently (hence "/bin" and "/lib") -
whereas "usr"
was a bigger, slower, cheaper disk, where the less frequently-used
stuff was stored ("/usr/bin", home directories, etc)...
Funny how the karmic wheel turns; I was suffering from the above
architecture until the early 1990s - arguably we still suffer from it
today, watch Perl building some time - and now I am redesigning the
same thing but at least now the whole OS squeezes into the small disk
pretty easily. :-)
As an aside there is nothing wrong with using ZFS - eg: a "zvol" - as
a
swap device; but just as you say, if we use real disks for root then
they will be so big that there''s probably no point in pushing swap off
to ZFS.
-a
--
Alec Muffett
http://www.google.com/search?q=alec-muffett
Alec Muffett wrote:> As I understand matters, from my notes to design the "perfect" home > NAS server :-) > > 1) you want to give ZFS entire spindles if at all possible; that will > mean it can enable and utilise the drive''s hardware write cache > properly, leading to a performance boost. You want to do this if you > can. Alas it knocks out the "split all disks into 7 & 493Gb > partitions" design concept. > > 2) I''ve considered pivot-root solutions based around a USB stick or > drive; cute, but I want a single tower box and no "dongles" > > 3) This leads me to the following design points: > > - enormous tower case with 10+ baysA good alternative is a smaller case with 6 bays and two 5 way SuperMicro cages. Better for space and drive cooling. Ian
> I definitely [i]don''t[/i] want to use flash for swap...You could use a ZVOL on the RAID-Z. Ok, not the most efficient thing, but there''s no sort of flag to disable parity on a specific object. I wish there was, exactly for this reason. -mg -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 648 bytes Desc: This is a digitally signed message part URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070615/66ef1b53/attachment.bin>
Hi Rick,> Hmm. Not sure I can do RAID5 (and boot from it). Presumably, though, > this would continue to function if a drive went bad. > > It also prevents ZFS from managing the devices itself, which I think > is undesirable (according to the ZFS Admin Guide). > > I''m also not sure if I have RAID5 support in the BIOS. I think it''s > just RAID0/1.Just a mainboard BIOS RAID is only like a software raid - accept that you may can''t get your data back if the RAID-controller die ... and when, you''ll need the same hardware again. So you may want to use a software-RAID instead of a plugin-card which supports buffering the read/write operations on power lost - it is nearly the same as when you use your BIOS (not a RAID-Controller) for that. And I RAID5 you''ll able to boot if you have a /boot partition outside the RAID5. Greetings Cyron -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070615/b03c62d4/attachment.bin>
Ian Collins wrote:> Alec Muffett wrote: >> As I understand matters, from my notes to design the "perfect" home >> NAS server :-) >> >> 1) you want to give ZFS entire spindles if at all possible; that will >> mean it can enable and utilise the drive''s hardware write cache >> properly, leading to a performance boost. You want to do this if you >> can. Alas it knocks out the "split all disks into 7 & 493Gb >> partitions" design concept. >> >> 2) I''ve considered pivot-root solutions based around a USB stick or >> drive; cute, but I want a single tower box and no "dongles" >> >> 3) This leads me to the following design points: >> >> - enormous tower case with 10+ bays> A good alternative is a smaller case with 6 bays and two 5 way > SuperMicro cages. Better for space and drive cooling.>> - HE/high-efficency mobo with 8+ SATA capabilityWhat 8-port-SATA motherboard models are Solaris-friendly? I''ve hunted and hunted and have finally resigned myself to getting a "generic" motherboard with PCIe-x16 and dropping in an Areca PCIe-x8 RAID card (in JBOD config, of course). As for drive arrangement, I went with the Addonics 5-drives-in-3-bays cages, rather similar to the SuperMicro ones mentioned above. -- Internet: windsor at warthog.com __o Life: Rob at Carrollton.Texas.USA.Earth _`\<,_ (_)/ (_) "They couldn''t hit an elephant at this distance." -- Major General John Sedgwick
comments from the peanut gallery... Rob Windsor wrote:> Ian Collins wrote: >> Alec Muffett wrote: >>> As I understand matters, from my notes to design the "perfect" home >>> NAS server :-) >>> >>> 1) you want to give ZFS entire spindles if at all possible; that will >>> mean it can enable and utilise the drive''s hardware write cache >>> properly, leading to a performance boost. You want to do this if you >>> can. Alas it knocks out the "split all disks into 7 & 493Gb >>> partitions" design concept.Most mobos still have IDE ports where you can hang a 40GByte disk or two for installing bootable ZFS (until it gets fully integrated into install) and as a dump device.>>> 2) I''ve considered pivot-root solutions based around a USB stick or >>> drive; cute, but I want a single tower box and no "dongles" >>> >>> 3) This leads me to the following design points: >>> >>> - enormous tower case with 10+ bays > >> A good alternative is a smaller case with 6 bays and two 5 way >> SuperMicro cages. Better for space and drive cooling. > >>> - HE/high-efficency mobo with 8+ SATA capability > > What 8-port-SATA motherboard models are Solaris-friendly? I''ve hunted > and hunted and have finally resigned myself to getting a "generic" > motherboard with PCIe-x16 and dropping in an Areca PCIe-x8 RAID card (in > JBOD config, of course).In the short term, look for AHCI (eg. Intel ICH6 and Via vt8251) for onboard SATA. NVidia SATA (nv_sata) is still not integrated :-(. -- richard
Alec Muffett wrote:> As I understand matters, from my notes to design the "perfect" home NAS > server :-) > > 1) you want to give ZFS entire spindles if at all possible; that will > mean it can enable and utilise the drive''s hardware write cache > properly, leading to a performance boost. You want to do this if you > can. Alas it knocks out the "split all disks into 7 & 493Gb partitions" > design concept.If you know that a disk has no consumers that require the write cache to be off (eg, UFS), then you can safely manually enable the write cache. ZFS always issues the cache flush command. So you could partition a (or several) disks into 1GB swap + 7GB boot zpool + 492GB data zpool, and then safely manually enable the write cache (with format -e, I believe). Just beware, do not enable the write cache if you have UFS partitions on the disk! --matt
On 6/15/07, Ian Collins <ian at ianshome.com> wrote:> Alec Muffett wrote: > > 2) I''ve considered pivot-root solutions based around a USB stick or > > drive; cute, but I want a single tower box and no "dongles"You could buy a laptop disk, or mount one of these on the motherboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16822998003 with a card like http://www.newegg.com/Product/Product.aspx?Item=N82E16820214113 for about $20.> A good alternative is a smaller case with 6 bays and two 5 way > SuperMicro cages. Better for space and drive cooling.Interesting in this regard is the yy-0221: http://www.directron.com/yy0221bk.html with 10 3.5" bays (well-cooled, too - fans mounted in front of 8 of them) and 6 5.25" bays. This doesn''t leave much room for a power supply, unfortunately - the Supermicro bays are almost 10" deep, and the case is only 18" deep. I''ll measure when I get home, but suffice it to say that the Enermax EG651P-VE I have in mine (at 140mm deep, if the manufacturer specs are correct) is a little on the tight side. But powerful PSUs aren''t necessarily any deeper - the Silverstone OP750, for example, is only 150mm, which I think would fit fine. Will
Here''s a start for a suggested equipment list: Lian Li case with 17 drive bays (12 3.5" , 5 5.25") http://www.newegg.com/Product/Product.aspx?Item=N82E16811112064 Asus M2N32-WS motherboard has PCI-X and PCI-E slots. I''m using Nevada b64 for iSCSI targets: http://www.newegg.com/Product/Product.aspx?Item=N82E16813131026 Your choice of CPU and memory. I''m using an Opteron 1212 http://www.newegg.com/Product/Product.aspx?Item=N82E16819105016 and DDR2-800 memory http://www.newegg.com/Product/Product.aspx?Item=N82E16820145034. BTW, I''m not spamming for Newegg, it''s just who I used and had the links handy ;^] TK This message posted from opensolaris.org
Tom Kimes wrote:> Here''s a start for a suggested equipment list: > > Lian Li case with 17 drive bays (12 3.5" , 5 5.25") http://www.newegg.com/Product/Product.aspx?Item=N82E16811112064 >So it only has room for one power supply. How many disk drives will you be installing? It''s not the steady state current that matters, as much as it is the ability to handle the surge current of starting to spin 17 disks from zero rpm. That initial surge can stall a lot of lesser power supplies. Will be interesting to see what happens here.> Asus M2N32-WS motherboard has PCI-X and PCI-E slots. I''m using Nevada b64 for iSCSI targets: > http://www.newegg.com/Product/Product.aspx?Item=N82E16813131026 > > Your choice of CPU and memory. > > I''m using an Opteron 1212 > http://www.newegg.com/Product/Product.aspx?Item=N82E16819105016 > > and DDR2-800 memory > http://www.newegg.com/Product/Product.aspx?Item=N82E16820145034. > > BTW, I''m not spamming for Newegg, it''s just who I used and had the links handy ;^] > > TK > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Brian Hechinger
2007-Jun-15 21:34 UTC
[zfs-discuss] Re: Karma Re: Re: Best use of 4 drives?
On Fri, Jun 15, 2007 at 02:27:18PM -0700, Neal Pollack wrote:> > So it only has room for one power supply. How many disk drives will you > be installing? > It''s not the steady state current that matters, as much as it is the > ability to handle the surge current > of starting to spin 17 disks from zero rpm. That initial surge can > stall a lot of lesser power supplies. > Will be interesting to see what happens here.Hmmm, that''s an interesting point. I remember the old days of having to stagger startup for large drives (physically large, not capacity large). Can that be done with SATA? -brian -- "Perl can be fast and elegant as much as J2EE can be fast and elegant. In the hands of a skilled artisan, it can and does happen; it''s just that most of the shit out there is built by people who''d be better suited to making sure that my burger is cooked thoroughly." -- Jonathan Patschke
Rob Windsor wrote:> > What 8-port-SATA motherboard models are Solaris-friendly? I''ve hunted > and hunted and have finally resigned myself to getting a "generic" > motherboard with PCIe-x16 and dropping in an Areca PCIe-x8 RAID card > (in JBOD config, of course). >I don''t know about 8 port SATA, but I used an Asus L1N64-SLI which has 12 and is very Solaris friendly. Ian
On 6/15/07, Brian Hechinger <wonko at 4amlunch.net> wrote:> Hmmm, that''s an interesting point. I remember the old days of having to > stagger startup for large drives (physically large, not capacity large). > > Can that be done with SATA?I had to link 2 600w power supplies together to be able to power on 12 drives... I believe it is up to the controller (and possibly the drives) to support staggering. But it is allowed in SATA if the controller/drives support it.
I''m having a heckuva time posting to individual replies (keep getting exceptions). I have a 1U rackmount server with 4 bays. I don''t think there''s any way to squeeze in a small IDE drive, and I don''t want to reduce the swap transfer rate if I can avoid it. The machine has 4 500 GB SATA drives, 2 GB RAM, and an AMD Opteron 175 Denmark 2.2GHz CPU The machine itself is a "TYAN B2865G20S4H Industry 19" rack-mountable 1U chassis Barebone Server NVIDIA nForce4 Ultra Socket 939 AMD Opteron Up to 1 GHz Hyper-Transport link support FSB": http://www.newegg.com/product/product.asp?item=N82E16856152019 I''m afraid I don''t know much about the different peripheral controllers available in the PC world (I''m a Mac guy), so I don''t know if I''ve shot myself in the foot with what I''ve bought. Sadly, I think I''ll just waste 500 GB of space for now; don''t really see a better solution. I may have to bail on the whole effort if I can''t get all my other apps running on b65 (Java, Resin, MySQL, etc). This message posted from opensolaris.org
Rick Mann wrote:> I''m having a heckuva time posting to individual replies (keep getting exceptions). > > I have a 1U rackmount server with 4 bays. I don''t think there''s any way to squeeze in a small IDE drive, and I don''t want to reduce the swap transfer rate if I can avoid it. > > The machine has 4 500 GB SATA drives, 2 GB RAM, and an AMD Opteron 175 Denmark 2.2GHz CPU > > The machine itself is a "TYAN B2865G20S4H Industry 19" rack-mountable 1U chassis Barebone Server NVIDIA nForce4 Ultra Socket 939 AMD Opteron Up to 1 GHz Hyper-Transport link support FSB": > > http://www.newegg.com/product/product.asp?item=N82E16856152019For the time being, these SATA disks will operate in IDE compatibility mode, so don''t worry about the write cache. There is some debate about whether the write cache is a win at all, but that is another rat hole. Go ahead and split off some space for boot and swap. Put the rest in ZFS. Mirror for best all-around performance.> I''m afraid I don''t know much about the different peripheral controllers available in the PC world (I''m a Mac guy), so I don''t know if I''ve shot myself in the foot with what I''ve bought. > > Sadly, I think I''ll just waste 500 GB of space for now; don''t really see a better solution. I may have to bail on the whole effort if I can''t get all my other apps running on b65 (Java, Resin, MySQL, etc).Java and mysql are already integrated into Solaris. The only resin I use comes from trees, not software developers :-). -- richard
Richard Elling wrote:> For the time being, these SATA disks will operate in IDE compatibility mode, so > don''t worry about the write cache. There is some debate about whether the write > cache is a win at all, but that is another rat hole. Go ahead and split off some > space for boot and swap. Put the rest in ZFS. Mirror for best all-around performance.I assume you mean to dedicate one drive to boot/swap/upgrade, and the other three drives to ZFS. But I can''t mirror with only 3 drives, so I think RAIDZ is best, wouldn''t you agree? This message posted from opensolaris.org
Rick Mann wrote:> Richard Elling wrote: > > >> For the time being, these SATA disks will operate in IDE compatibility mode, so >> don''t worry about the write cache. There is some debate about whether the write >> cache is a win at all, but that is another rat hole. Go ahead and split off some >> space for boot and swap. Put the rest in ZFS. Mirror for best all-around performance. >> > > I assume you mean to dedicate one drive to boot/swap/upgrade, and the other three drives to ZFS. But I can''t mirror with only 3 drives, so I think RAIDZ is best, wouldn''t you agree? > >Try some tests on you box: slice one drive for root, swap and the rest ZFS. Install on that drive and create a raidz pool with the other drives, benchmark. Slice the other drives in the same way as the first and build either a raidz pool or a stripe of two of mirrors from the four ZFS slices, benchmark. The time will be well spent. Ian
Rick Mann wrote:> Richard Elling wrote: > >> For the time being, these SATA disks will operate in IDE compatibility mode, so >> don''t worry about the write cache. There is some debate about whether the write >> cache is a win at all, but that is another rat hole. Go ahead and split off some >> space for boot and swap. Put the rest in ZFS. Mirror for best all-around performance. > > I assume you mean to dedicate one drive to boot/swap/upgrade, and the other three drives to ZFS. But I can''t mirror with only 3 drives, so I think RAIDZ is best, wouldn''t you agree?What I would do: 2 disks: slice 0 & 3 root (BE and ABE), slice 1 swap/dump, slice 6 ZFS mirror 2 disks: whole disk mirrors The ZFS config would be a dynamic stripe of mirrors. Later, when you spring for 1TByte disks, you can replace them one at a time, and grow with minimal effort. KISS Your challenge will be how to back this beast up. -- richard
rmann at latencyzero.com said:> Richard Elling wrote: >> For the time being, these SATA disks will operate in IDE compatibility mode, >> so don''t worry about the write cache. There is some debate about whether >> the write cache is a win at all, but that is another rat hole. Go ahead >> and split off some space for boot and swap. Put the rest in ZFS. Mirror >> for best all-around performance. > > I assume you mean to dedicate one drive to boot/swap/upgrade, and the other > three drives to ZFS. But I can''t mirror with only 3 drives, so I think RAIDZ > is best, wouldn''t you agree?I''ll chime in here with a "me too" to Richard''s suggestion, though slightly different layout. We have a Sun T2000 here with 4x 73GB drives, and it works just fine to mix UFS and ZFS on old-fashioned slices (partitions) across all four of them. On your first disk, use s0 for a large-enough root, maybe 10GB; Then s1 is swap; The rest of the disk can be s6, which you''ll use for ZFS. Now, slice up all four disks exactly the same way. Create an SVM mirror across the first two s0''s, that''s your root. Create a 2nd SVM mirror across the first two s1''s, that''s your swap. The 3rd and 4th s0''s and s1''s can be anything you like, maybe mirrored alternate boot env. for liveupgrade, etc. I made mine into a ZFS-mirrored /export. Lastly, use the four s6''s to create a "big" RAID-Z pool. With your four 500GB drives, you''ve given up only 12-14GB each for your system usage, so the remaining 486-488GB should give you nearly 1.4TB of useable RAID-Z protected space. Sure, it''s not "optimal", but it''s really quite good. Regards, Marion
Richard Elling wrote:> What I would do: > 2 disks: slice 0 & 3 root (BE and ABE), slice 1 swap/dump, slice 6 > ZFS mirror > 2 disks: whole disk mirrors >I don''t understand "slice 6 zfs mirror". A mirror takes *two* things of the same size. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/dd-b Pics: http://dd-b.net/dd-b/SnapshotAlbum, http://dd-b.net/photography/gallery Dragaera: http://dragaera.info
David Dyer-Bennet wrote:> Richard Elling wrote: >> What I would do: >> 2 disks: slice 0 & 3 root (BE and ABE), slice 1 swap/dump, slice >> 6 ZFS mirror >> 2 disks: whole disk mirrors >> > I don''t understand "slice 6 zfs mirror". A mirror takes *two* things > of the same size. >Note the "2 disks:". Ian
On 6/15/07, Brian Hechinger <wonko at 4amlunch.net> wrote:> On Fri, Jun 15, 2007 at 02:27:18PM -0700, Neal Pollack wrote: > > > > So it only has room for one power supply. How many disk drives will you > > be installing? > > It''s not the steady state current that matters, as much as it is the > > ability to handle the surge current > > of starting to spin 17 disks from zero rpm. That initial surge can > > stall a lot of lesser power supplies. > > Will be interesting to see what happens here.Drives only really take 1.5A or so from the 12V rail when spinning up, but it''s a good rule of thumb to pretend they take 3A each on top of the other junk in your system. A low-usage system like a Core 2 Duo with a non-nV chipset and onboard or low-end video can run in 100 watts with no problems, so add 51A of 12V rail capacity to 100 watts worth and you can still find PSUs that supply that. If you do staggered spinup, you can allocate something more like 10A plus one amp per drive. In either case, an OP1000 (http://www.newegg.com/Product/Product.aspx?Item=N82E16817256010) or a PCP&C TC1KW-SR (http://www.newegg.com/Product/Product.aspx?Item=N82E16817703007) would do you even if you wanted to do SLI or something equally ridiculous.> Hmmm, that''s an interesting point. I remember the old days of having to > stagger startup for large drives (physically large, not capacity large). > > Can that be done with SATA?Can and is. On my Marvell 88sx6081 controller, it happened without my having to configure anything magical. Will
Ian Collins wrote:> David Dyer-Bennet wrote: >> Richard Elling wrote: >>> What I would do: >>> 2 disks: slice 0 & 3 root (BE and ABE), slice 1 swap/dump, slice >>> 6 ZFS mirror >>> 2 disks: whole disk mirrors >>> >> I don''t understand "slice 6 zfs mirror". A mirror takes *two* things >> of the same size. >> > Note the "2 disks:".Yeah, should probably draw a picture :-) -- richard
Ian Collins wrote:> Rob Windsor wrote: >> What 8-port-SATA motherboard models are Solaris-friendly? I''ve hunted >> and hunted and have finally resigned myself to getting a "generic" >> motherboard with PCIe-x16 and dropping in an Areca PCIe-x8 RAID card >> (in JBOD config, of course).> I don''t know about 8 port SATA, but I used an Asus L1N64-SLI which has > 12 and is very Solaris friendly.Good lord! That''s a $400 mobo and CPUs are $400/ea (give or take a few bucks, depending on where you shop). My solution was cheaper and more upgrade-friendly. :) Rob++ -- |Internet: windsor at warthog.com __o |Life: Rob at Carrollton.Texas.USA.Earth _`\<,_ | (_)/ (_) |"They couldn''t hit an elephant at this distance." | -- Major General John Sedgwick
Rob Windsor wrote:> Ian Collins wrote: >> Rob Windsor wrote: >>> What 8-port-SATA motherboard models are Solaris-friendly? I''ve hunted >>> and hunted and have finally resigned myself to getting a "generic" >>> motherboard with PCIe-x16 and dropping in an Areca PCIe-x8 RAID card >>> (in JBOD config, of course). > >> I don''t know about 8 port SATA, but I used an Asus L1N64-SLI which has >> 12 and is very Solaris friendly. > > Good lord! > > That''s a $400 mobo and CPUs are $400/ea (give or take a few bucks, > depending on where you shop). > > My solution was cheaper and more upgrade-friendly. :) >But not as fast or anywhere near as good at drying clothes! Ian
On 6/16/07, Rob Windsor <windsor at warthog.com> wrote:> Good lord! > > That''s a $400 mobo and CPUs are $400/ea (give or take a few bucks, > depending on where you shop). > > My solution was cheaper and more upgrade-friendly. :)Remind me again, how much do Areca cards cost? Then again, the FX platform is pretty awful in terms of power usage/heat dissipation. Something like this might be more interesting: http://www.newegg.com/product/Product.asp?item=N82E16813131026 especially since Newegg''s currently giving away a 3600+ with that board. For $300, you get 3 onboard sata + 1 external, on a Marvell chipset (which should work under Solaris just fine), 2 64/133 PCI-X slots, and 2 real x16 slots. Sounds pretty good to me. Add $130 for 2 gigs of memory: http://www.newegg.com/Product/Product.aspx?Item=N82E16820134113 and you''ve got yourself a decent platform to upgrade from - 2 of those nice Supermicro controllers, for example, will fit fine. My $.02 Will
Will Murnane wrote:> On 6/16/07, Rob Windsor <windsor at warthog.com> wrote: >> Good lord! >> >> That''s a $400 mobo and CPUs are $400/ea (give or take a few bucks, >> depending on where you shop). >> >> My solution was cheaper and more upgrade-friendly. :) > > Remind me again, how much do Areca cards cost?$60 for mobo, $60 for cpu, $150 for memory, $530 for 8-port Areca card. :) I was shooting for a socket-939 since I already had a spare 3800+ and memory.> Something like this might be more interesting: > http://www.newegg.com/product/Product.asp?item=N82E16813131026Yes, that''s pretty tasty for an AM2.> 2 of those > nice Supermicro controllers, for example, will fit fine.I''m assuming you''re referring to AOC-SAT2-MV8 (only 3Gb/s card listed). I don''t see those listed in the HCL as either "Supermicro" or "Marvell", which is why they weren''t even a consideration. Rob++ -- |Internet: windsor at warthog.com __o |Life: Rob at Carrollton.Texas.USA.Earth _`\<,_ | (_)/ (_) |"They couldn''t hit an elephant at this distance." | -- Major General John Sedgwick
On 6/16/07, Rob Windsor <windsor at warthog.com> wrote:> $60 for mobo, $60 for cpu, $150 for memory, $530 for 8-port Areca card. :)Yeah, I knew how much they cost ;) Google still seems to be up. I was merely pointing out that spending $500 on the mobo and getting a $100 card or spending $100 on the mobo and $500 on the Areca card are roughly equal, when it comes down to it.> I don''t see those listed in the HCL as either "Supermicro" or "Marvell", > which is why they weren''t even a consideration.They are. They''re based on the Marvell 88sx6081, which is indeed supported - it''s actually the same controller as in the Areca cards, and it''s also used in the Thumpers. I wish someone would make a card with a pci-express to pci-x bridge chip and two of those controllers on it. Hmm, I just realized I''m describing the highpoint 2340: http://www.highpoint-tech.com/USA/rr2340.htm I dunno if it''s supported under Solaris. Perhaps I can find someone to boot Belenix and let me know if it works. Will
Will Murnane wrote:> On 6/16/07, Rob Windsor <windsor at warthog.com> wrote:>> I don''t see those listed in the HCL as either "Supermicro" or "Marvell", >> which is why they weren''t even a consideration. > They are. They''re based on the Marvell 88sx6081, which is indeed > supported - it''s actually the same controller as in the Areca cards, > and it''s also used in the Thumpers.I still don''t see them in the HCL. *Very* frustrating. After getting burned nicely back in the 2.6 days, I don''t even go shopping until I find it in the HCL.> Hmm, I just realized I''m describing the highpoint 2340: > http://www.highpoint-tech.com/USA/rr2340.htm I dunno if it''s > supported under Solaris. Perhaps I can find someone to boot Belenix > and let me know if it works.Is the HighPoint 2320 supported? (again, can''t find -that- in the HCL, either!) That''s half the price of the Areca. Rob++ -- |Internet: windsor at warthog.com __o |Life: Rob at Carrollton.Texas.USA.Earth _`\<,_ | (_)/ (_) |"They couldn''t hit an elephant at this distance." | -- Major General John Sedgwick
On 6/16/07, Rob Windsor <windsor at warthog.com> wrote:> I still don''t see them in the HCL. *Very* frustrating.http://www.sun.com/bigadmin/hcl/driverlist.html Search for "6081" in that page.> After getting burned nicely back in the 2.6 days, I don''t even go > shopping until I find it in the HCL.Probably still a good idea. I bought my stuff with Linux in mind, and then discovered that Solaris worked better than Linux did with it.> Is the HighPoint 2320 supported? (again, can''t find -that- in the HCL, > either!) That''s half the price of the Areca.I don''t know, but I''m asking at a forum where I know several 2320 owners are around if someone will test for me. It''d be a heck of a convenient piece of hardware - half the cost of the Areca, or twice the ports for the same price. I''ll be sure to revive this thread if I get it tested. Will
On Fri, 15 Jun 2007, Will Murnane wrote:> On 6/15/07, Ian Collins <ian at ianshome.com> wrote: >> Alec Muffett wrote: >> > 2) I''ve considered pivot-root solutions based around a USB stick or >> > drive; cute, but I want a single tower box and no "dongles" > You could buy a laptop disk, or mount one of these on the motherboard: > http://www.newegg.com/Product/Product.aspx?Item=N82E16822998003 with a > card like http://www.newegg.com/Product/Product.aspx?Item=N82E16820214113 > for about $20. >> A good alternative is a smaller case with 6 bays and two 5 way >> SuperMicro cages. Better for space and drive cooling. > Interesting in this regard is the yy-0221: > http://www.directron.com/yy0221bk.html with 10 3.5" bays (well-cooled, > too - fans mounted in front of 8 of them) and 6 5.25" bays. This > doesn''t leave much room for a power supply, unfortunately - the > Supermicro bays are almost 10" deep, and the case is only 18" deep. > I''ll measure when I get home, but suffice it to say that the Enermax > EG651P-VE I have in mine (at 140mm deep, if the manufacturer specs are > correct) is a little on the tight side. But powerful PSUs aren''t > necessarily any deeper - the Silverstone OP750, for example, is only > 150mm, which I think would fit fine.Speaking of cases, I built a system using the Antec P180 - a very unusual case with great disk capacity and cooling. It has since been improved - with the introduction of the P182. Here is a good review (lots of images): http://www.hi-techreviews.com/reviews_2007/Antec182/P.htm There is one misleading image here: http://hi-techreviews.com/reviews_2007/Antec182/IMG_2442.JPG Note how the main motherboard power wiring harness comes up directy from the power supply compartment. That is how the P180 was designed - the P182 allows for this wiring harness (and other cables BTW) to be routed on the side of the case, i.e., on the motherboard mounting plate - you''ll see the large cable ties intended for this here: http://hi-techreviews.com/reviews_2007/Antec182/IMG_2416.JPG And a correctly configured P182 build here: http://www.xsreviews.co.uk/reviews/cases/antec-p182/4/ Downsides: - relatively expensive - comes with no power supply - physically large Upsides: - great cooling - allows building a quiet, office friendly, system Regards, Al Hopper Logical Approach Inc, Plano, TX. al at logical-approach.com Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007 http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
I am currently using 6 drives and a 550W power supply, so I''m not pushing the hardware at this point. I do understand your point. However, if you are willing to mod the case, there is room for a second power supply above where the primary p/s mounts. The case modification should be fairly straight forward to create the mount screw holes and fan vent above the factory cuts. A steady hand and few minutes with a Dremel should do the trick. If you must have case that is setup for dual p/s, a Supermicro case is probably your best option though it will be a lot more $$$. This message posted from opensolaris.org
On 6/15/07, Will Murnane <will.murnane at gmail.com> wrote:> Interesting in this regard is the yy-0221: > http://www.directron.com/yy0221bk.html with 10 3.5" bays (well-cooled, > too - fans mounted in front of 8 of them) and 6 5.25" bays. This > doesn''t leave much room for a power supply, unfortunately - the > Supermicro bays are almost 10" deep, and the case is only 18" deep. > I''ll measure when I get home, but suffice it to say that the Enermax > EG651P-VE I have in mine (at 140mm deep, if the manufacturer specs are > correct) is a little on the tight side.I''ve broken out the measuring tape as promised, and it isn''t as desperately tight as I remembered. The case is 16 inches deep, not counting the plastic front cover, the Supermicro cages take up almost exactly 9 inches of that to the back of the fan. The Enermax power supply is actually 145mm deep by my measurements, not 140 as claimed, and this leaves about an inch and a quarter between the fan and the power supply. With some creative fan replacement or a small power supply, it''s not too bad. Will
On June 16, 2007 7:37:31 PM -0400 Will Murnane <will.murnane at gmail.com> wrote:> I wish someone would make a card with a pci-express to pci-x bridge > chip and two of those controllers on it. Hmm, I just realized I''m > describing the highpoint 2340: > http://www.highpoint-tech.com/USA/rr2340.htmeh, sorry but where is the pci-e to pci-x bridge on this card? -frank