Hi, I have a small Linux server PC at home (Intel Core2 Q9300, 4 GB RAM), and I''m seriously considering switching to OpenSolaris (Indiana, 2008.11) in the near future, mainly because of ZFS. The idea is to run the existing CentOS 4.7 system inside a VM and let it NFS mount home directories and other filesystems from OpenSolaris. I might migrate more services from Linux over time, but for now, the filesystems are priority one. Since most of my questions are actually about ZFS, I thought I''d ask here directly. First of all, I''m on a budget and while "cheap" is important, "value for the money" is critical. So the less number of disk, the better (not only because of price but also because of power consumption and the lack of cooling). This also means that I prefer RAID-Z to mirrors for less critical data. 1) My plan is to install 2008.11 on three 1 TB disks (Samsung SpinPoint F1, two new and one currently containing Linux data). I will start with a single, new disk and install the OS on a zpool inside an fdisk partition, say 100 GB large. This is where the OS and /home will live. The rest of the disk will later be used for less critical data (music, video, VMs, backups, etc), inside an fdisk partition of type "Unix". Once installed, I''ll attach the second new disk, using identical partition layout, and attach the 100 GB partition as mirror to the root pool. I''ll then create a sparse ZFS volume and use this together with the unused fdisk partition on each disk to create a 3-way RAID-Z pool, and finally degrade it by taking the sparse file offline. I''ll then start migrating the Linux files, probably using a VM directly mounting the ext3 filesystems from the old 1 TB disk and copying home directories to the mirror pool and media files to the raid-z pool. Finally, I''ll reformat and attach the third disk to the pools. I thus hope to end up with three disk and two pools: one small 3-way mirror for critical data and one large 3-way raid-z pool for the rest. How does this idea sound to you? Will I be able to enable the write cache in this setup (not that write speed matters much to me, but still)? 2) Given the perhaps unusual disk layout, do you think I''ll run into trouble concerning OS upgrades or if/when one of the disks fails? I''ve tried the procedure in VMware and found a few gotchas, but nothing too serious (like "zpool import -f rpool" to make grub work after installation, and trying to create the second zpool on c4t0d0p1 -- which happened to be the same as c4t0d0s0, where rpool lives -- instead of c4t0d0p2; funny "zpool create" didn''t complain?) 3) One problem I don''t understand why I got is this: When I attach a new virgin disk to the system, I first run format->fdisk to make two fdisk partitions and then use prtvtoc/fmthard to create the splices on the solaris partition. When I then try to attach the new c4t1d0s0 to the existing c4t1d0s0, zpool complains that only complete disks can be attached. However, after a reboot, the slice attaches without problems (except that I have to use -f since c4t1d0s0 overlaps with c4t1d0s2, which is also something I don''t understand -- of course it does?). How come? 4) I plan to use a sparse ZFS volume for the Linux VM root disk, probably from the mirror pool. Objections? 5) Given that this is all cheap PC hardware ... can I move a disk from a broken controller to another, and if so, how? I tried this in VMware, but could not figure out how to re-attach the moved disk. zpool complains that the moved disk is part of an active zpool and -f didn''t help at all. Any input would be greatly appreciated! -- ---- Martin Blom --------------------------- martin at blom.org ---- Eccl 1:18 http://martin.blom.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3284 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081116/8e19cefe/attachment.bin>
I don''t know much about working with slices in Solaris I''m afraid, but to me that sounds like a pretty good setup for a home server, and I can''t see why the layout would cause you any problems. In theory you''ll be able to swap controllers without any problems too. That''s one of the real benefits of ZFS for me, although it''s not something I''ve had to put into practice yet. -- This message posted from opensolaris.org
On Sun, 16 Nov 2008 03:13:59 PST Ross <myxiplx at googlemail.com> wrote:> I don''t know much about working with slices in Solaris I''m afraid, > but to me that sounds like a pretty good setup for a home server, and > I can''t see why the layout would cause you any problems. > > In theory you''ll be able to swap controllers without any problems > too. That''s one of the real benefits of ZFS for me, although it''s > not something I''ve had to put into practice yet.One minor issue: zfs works best on whole disks. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv101 ++ + All that''s really worth doing is what we do for others (Lewis Carrol)
Well yes, but he doesn''t sound too worried about performance, and I''m not aware of any other issues with splitting drives? And if you did want performance later, it would probably be possible to add a flash drive for cache once the prices drop. -- This message posted from opensolaris.org
Miles Nordin wrote:> > mb> 5) Given that this is all cheap PC hardware ... can I move a > mb> disk from a broken controller to another > > zpool export, zpool import. >I was testing with the rpool, but "zpool import -f" when booting for the CD did the trick. Thanks for the hint.> If the pool is only DEGRADED it would be nice to do it online, but I > don''t know a way to do that. > > mb> How does this idea sound to you? > > I think you need a backup in a separate pool or a non-ZFS filesystem. > The backup could be .tar files or an extracted rsync copy, but somehow > I think you need protection against losing the whole pool to software > bugs or operator error. There are other cases where you might want to > destroy and recreate the pool, like wanting to remove a slog or change > the raidz/raidz2/mirror level, but I think that''s not why you need it. > You need it for protection. losing the pool is really possible. >I do intend to keep backups both before and after, but are you referring to the actual migration or when everything is completed? I know the data is at risk while transferring the old content and when attaching the third drive; what I''m worried about is if I''m risking it more than usual when the procedure is done? -- ---- Martin Blom --------------------------- martin at blom.org ---- Eccl 1:18 http://martin.blom.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3284 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081116/e19b369c/attachment.bin>
On Sun, 16 Nov 2008, Ross wrote:> Well yes, but he doesn''t sound too worried about performance, and > I''m not aware of any other issues with splitting drives?Besides some possible loss of performance, splitting drives tends to blow natural redundancy models where you want as little coupling as possible. If the part of the drive that zfs is using fails, you don''t want to have to worry about whether the data in other partitions is still recoverable so maybe you should delay repair of the zfs pool while you investigate the rest of the drive. You just want to slap in a new drive and wait for the pool to recover as quickly as possible. It is best to minimize commonality between redundant components to the maximum extent possible. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
>>>>> "mb" == Martin Blom <martin at blom.org> writes:mb> if I''m risking it more than usual when the procedure is done? yeah, that is my opinion: when the procedure is done, using ZFS without a backup is risking the data more than using UFS or ext3 without a backup. Is that a clear statement? I can ramble on, but maybe that''s all you care to hear. ZFS or not, I''ve become a big believer that filesystem-level backup is always important for data that must last a decade, not just RAID or snapshots, so it doesn''t mean don''t use ZFS, it means this is the time to start building proper backup into your budget. With huge amounts of data, you can get into a situation where you need to make a copy of the data, and you''ve nowhere to put it and no time and money to create a space for it, and you find yourself painted into a corner. This is not as much the case if you''ve bought a single big external drive---you can afford to buy another drive, and the new drive works instantly---but with big RAIDs you have to save/budget/build to make space to copy them. Why would you suddenly need a copy? Well, maybe you need to carry the data with you to deliver it to someone else. You risk damaging the copy you''re physically transporting, so you should always have a stationary copy too. Maybe you need to (I''m repeating myself again so maybe just reread my other post) change the raid stripe arrangement (ex., widen the stripe when you add a fourth disk, otherwise you end up stuck with raidz(3disk) * 2 when you could have the same capacity and more protection with raidz2(6disk) * 1), or remove a slog, or work around a problem by importing the pool on an untrustworthy SXCE release or hacked zfs code that might destroy everything, or you want to test-upgrade the pool to a new version to see if it fixes a problem but think you might want to downgrade it to the old zpool version if you run into other problems. Without a copy, you will be so fearful of every reasonable step you take, you will make prudish decisions and function slowly. The possible exception to needing backups is these two-level filesystems like GlusterFS or googlefs or maybe samfs? These are mid-layer filesystems that are backed by ordinary filesystems beneath them, not block devices, and they replicate the data across the ordinary filesystems. Some of these may have worst-case recovery schemes that are pretty good, especially if you have a few big files rather than many tiny ones so you can live with getting just the insides of the file back like fsck gives you in lost+found. And they don''t use RAID-like parity/FEC schemes, rather they only make mirror-like copies of the files, and they''ve usually the ability to evacuate an underlying filesystem, so you''re less likely to be painted into a corner like i described---you always own enough disk for two or three complete copies. but I don''t have experience here---that''s my point. Maybe still for these, maybe not, but at least for all block-backed filesystems, my experience says that you need a backup, because within a decade you''ll make a mistake or hit a corruption bug or need a copy. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081116/9d09db41/attachment.bin>
Miles Nordin wrote:> mb> if I''m risking it more than usual when the procedure is done? > > yeah, that is my opinion: when the procedure is done, using ZFS > without a backup is risking the data more than using UFS or ext3 > without a backup. Is that a clear statement? > > > I can ramble on, but maybe that''s all you care to hear. >Thanks all for your input. I guess basically the idea is sound for my needs; however, Miles'' words made me do my homework and read the mail archives. So I don''t know ... The idea of loosing a complete zpool just because the power goes out (which does happen once of twice a year here), or simply because of the fact that I will be running on toy hardware, is really not comfortable. I''m quite confident ext3 will never do that to me. In a mail from last month Jeff Bonwick wrote on this list that he''s working on better recovery from inconsistent filesystems. I guess that''s something I should wait for. -- ---- Martin Blom --------------------------- martin at blom.org ---- Eccl 1:18 http://martin.blom.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3284 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081118/990e34f3/attachment.bin>
To be honest, I don''t think it''s that much of a risk: We get regular power cuts at home (probably 8-10 a year), and I''ve got a solaris box that was originally snv_70, and is now snv_98 that''s survived over half a dozen without any issues at all. It wasn''t ZFS boot originally, but has been upgraded to that now and has coped with another three power cuts since then without any problems. I trust this box enough that I don''t even have a keyboard, monitor or mouse attached to it. I just hit the on button once I think the power is stable again, and wait for my CIFS and NFS shares to reappear. I don''t even think I''m using mirrored boot volumes on this, I''m using raid-z2 for the data and it''s only half an hours work to re-install if a disk dies. -- This message posted from opensolaris.org