I''ve been playing around with zfs root pool mirroring and came across some problems. I have no problems mirroring the root pool if I have both disks attached during OpenSolaris installation (installer sees 2 disks). The problem occurs when I only have one disk attached to the system during install. After OpenSolaris installation completes, I attach the second disk and try to create a mirror but I cannot. Here are the steps I go through: 1) install OpenSolaris onto 16GB disk 2) after successful install, shutdown, and attach second disk (also 16GB) 3) fdisk -B 4) partition 5) zfs attach Step 5 fails, giving a "disk too small" error. What I noticed about the second disk is that it has a 9th partition called "alternates" that takes up about 15MBs. This partition doesn''t exist in the first disk and I believe is what''s causing the problem. I can''t figure out how to delete this partition and I don''t know why it''s there. How do I mirror the root pool if I don''t have both disks attached during OpenSolaris installation? I realize I can just use a disk larger than 16GBs, but that would be a waste. -- This message posted from opensolaris.org
Hi Noz, This problem was reported recently and this bug was filed: 6844090 zfs should be able to mirror to a smaller disk I believe slice 9 (alternates) is an older method for providing alternate disk blocks on x86 systems. Apparently, it can be removed by using the format -e command. I haven''t tried this though. I don''t think removing slice 9 will help though if these two disks are not identical, hence the bug. You can workaround this problem by attaching a slightly larger disk. Cindy noz wrote:> I''ve been playing around with zfs root pool mirroring and came across some problems. > > I have no problems mirroring the root pool if I have both disks attached during OpenSolaris installation (installer sees 2 disks). > > The problem occurs when I only have one disk attached to the system during install. After OpenSolaris installation completes, I attach the second disk and try to create a mirror but I cannot. > > Here are the steps I go through: > 1) install OpenSolaris onto 16GB disk > 2) after successful install, shutdown, and attach second disk (also 16GB) > 3) fdisk -B > 4) partition > 5) zfs attach > > Step 5 fails, giving a "disk too small" error. > > What I noticed about the second disk is that it has a 9th partition called "alternates" that takes up about 15MBs. This partition doesn''t exist in the first disk and I believe is what''s causing the problem. I can''t figure out how to delete this partition and I don''t know why it''s there. How do I mirror the root pool if I don''t have both disks attached during OpenSolaris installation? I realize I can just use a disk larger than 16GBs, but that would be a waste.
noz wrote:> I''ve been playing around with zfs root pool mirroring and came across some problems. > > I have no problems mirroring the root pool if I have both disks attached during OpenSolaris installation (installer sees 2 disks). > > The problem occurs when I only have one disk attached to the system during install. After OpenSolaris installation completes, I attach the second disk and try to create a mirror but I cannot. > > Here are the steps I go through: > 1) install OpenSolaris onto 16GB disk > 2) after successful install, shutdown, and attach second disk (also 16GB) > 3) fdisk -B > 4) partition >This is a critical step and it is important that you create a SMI label, not an EFI label. For the exact steps, please consult the ZFS Administration Guide which has a section on this very process. -- richard> 5) zfs attach > > Step 5 fails, giving a "disk too small" error. > > What I noticed about the second disk is that it has a 9th partition called "alternates" that takes up about 15MBs. This partition doesn''t exist in the first disk and I believe is what''s causing the problem. I can''t figure out how to delete this partition and I don''t know why it''s there. How do I mirror the root pool if I don''t have both disks attached during OpenSolaris installation? I realize I can just use a disk larger than 16GBs, but that would be a waste. >
> I believe slice 9 (alternates) is an older method for > providing > alternate disk blocks on x86 systems. Apparently, it > can be removed by > using the format -e command. I haven''t tried this > though.format -e worked!! It is resilvering as I type this message. Thanks!!> I don''t think removing slice 9 will help though if > these two disks > are not identical, hence the bug.They are identical though. The only difference is s0 on the second disk is slightly smaller than s0 on the first disk due to s9 stealing about 15MBs of space. So when I invoked "zpool attach -f rpool c7d0s0 c7d1s0", I get the "too small" error. After deleting s9, everything worked okay. Thanks Cindy!! -- This message posted from opensolaris.org
On 06/04/09 06:44 PM, Cindy.Swearingen at Sun.COM wrote:> Hi Noz, > > This problem was reported recently and this bug was filed: > > 6844090 zfs should be able to mirror to a smaller diskIs this filed on bugs or defects? I had the exact same problem, and it turned out to be a rounding error in Solaris format/fdisk. The only way I could fix it was to use Linux (well, Fedora) sfdisk to make both partitions exactly the same number of bytes. The alternates partition seems to be hard wired on older disks and AFAIK there''s no way to use that space. sfdisk is on the Fedora live CD if you don''t have a handy Linux system to get it from. BTW the disks were nominally the same size but had different geometries. Since I can''t find 6844090, I have no idea what it says, but this really seems to be a bug in fdisk, not ZFS, although I would think ZFS should be able to mirror to a disk that is only a tiny bit smaller... -- Frank> I believe slice 9 (alternates) is an older method for providing > alternate disk blocks on x86 systems. Apparently, it can be removed by > using the format -e command. I haven''t tried this though. > > I don''t think removing slice 9 will help though if these two disks > are not identical, hence the bug. > > You can workaround this problem by attaching a slightly larger disk. > > Cindy > > > noz wrote: >> I''ve been playing around with zfs root pool mirroring and came across >> some problems. >> >> I have no problems mirroring the root pool if I have both disks >> attached during OpenSolaris installation (installer sees 2 disks). >> >> The problem occurs when I only have one disk attached to the system >> during install. After OpenSolaris installation completes, I attach the >> second disk and try to create a mirror but I cannot. >> >> Here are the steps I go through: >> 1) install OpenSolaris onto 16GB disk >> 2) after successful install, shutdown, and attach second disk (also 16GB) >> 3) fdisk -B >> 4) partition >> 5) zfs attach >> >> Step 5 fails, giving a "disk too small" error. >> >> What I noticed about the second disk is that it has a 9th partition >> called "alternates" that takes up about 15MBs. This partition doesn''t >> exist in the first disk and I believe is what''s causing the problem. I >> can''t figure out how to delete this partition and I don''t know why >> it''s there. How do I mirror the root pool if I don''t have both disks >> attached during OpenSolaris installation? I realize I can just use a >> disk larger than 16GBs, but that would be a waste. > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
I''ve been dealing with this at an unusually high frequency these days. It''s even dodgier on SPARC. My recipe has been to run format -e and first try to label as SMI. Solaris PCs sometimes complain that the disk needs fdisk partitioning and I always delete *all* partitions, exit fdisk, enter fdisk again, at which time it prompts me to make it a 100% Solaris partition and I agree. I think that may have been my way of accomplishing the part where you bailed and did your Linux partition hacking, Frank. Anyway, after that, remember that you currently still have to use SMI labels for the root pool, so run label and choose that. format(1M) sometimes momentarily hangs and crashes the there, iirc -- if this happens to you, run it again and it''s fine the second time through. Maybe related to fdisk changes plus RAM disklabel vs on-disk disklabel, etc. Anyway, make slice 0 identical to slice 2, which is congruent with the OpenSolaris install procedure; however to eliminate human error, a prtvtoc /dev/dsk/cXtXd0s0 | fmthard -s- /dev/rdsk/cXtXd0s0 should work for you, at this point, to duplicate the label from the first disk to the second. Read and understand that command before issuing it as it does very little second guessing and can easily nuke stuff if you run it wrong. Note: on SPARC, if you''re using generic disks that happen to be the same part numbers as real Sun disks, but have generic labels instead of the nice "SUN146G" and the like, you can run into this issue of ZFS refusing to mirror to a smaller disk again. My solution to that was to run format => type and select the type matching the other disk, then continue such that all reasonably matching disks have the same type. This seems to help with geometry issues and also completely changed (drastically for the better) the performance characteristics of some disks in my SunFire v440. *Note to others who may end up here later or for tangential reasons: Realise that pretty much all of these gross manipulations of your disk configuration are sufficiently heavy-handed that you''re going to lose everything stored on the disk in question when you run them. hth jake Frank Middleton wrote:> On 06/04/09 06:44 PM, Cindy.Swearingen at Sun.COM wrote: >> Hi Noz, >> >> This problem was reported recently and this bug was filed: >> >> 6844090 zfs should be able to mirror to a smaller disk > > Is this filed on bugs or defects? I had the exact same problem, > and it turned out to be a rounding error in Solaris format/fdisk. > The only way I could fix it was to use Linux (well, Fedora) sfdisk > to make both partitions exactly the same number of bytes. The > alternates partition seems to be hard wired on older disks and > AFAIK there''s no way to use that space. sfdisk is on the Fedora > live CD if you don''t have a handy Linux system to get it from. > BTW the disks were nominally the same size but had different > geometries. > > Since I can''t find 6844090, I have no idea what it says, but this > really seems to be a bug in fdisk, not ZFS, although I would think > ZFS should be able to mirror to a disk that is only a tiny bit > smaller... > > -- Frank > >> I believe slice 9 (alternates) is an older method for providing >> alternate disk blocks on x86 systems. Apparently, it can be removed by >> using the format -e command. I haven''t tried this though. >> >> I don''t think removing slice 9 will help though if these two disks >> are not identical, hence the bug. >> >> You can workaround this problem by attaching a slightly larger disk. >> >> Cindy >> >> >> noz wrote: >>> I''ve been playing around with zfs root pool mirroring and came across >>> some problems. >>> >>> I have no problems mirroring the root pool if I have both disks >>> attached during OpenSolaris installation (installer sees 2 disks). >>> >>> The problem occurs when I only have one disk attached to the system >>> during install. After OpenSolaris installation completes, I attach the >>> second disk and try to create a mirror but I cannot. >>> >>> Here are the steps I go through: >>> 1) install OpenSolaris onto 16GB disk >>> 2) after successful install, shutdown, and attach second disk (also >>> 16GB) >>> 3) fdisk -B >>> 4) partition >>> 5) zfs attach >>> >>> Step 5 fails, giving a "disk too small" error. >>> >>> What I noticed about the second disk is that it has a 9th partition >>> called "alternates" that takes up about 15MBs. This partition doesn''t >>> exist in the first disk and I believe is what''s causing the problem. I >>> can''t figure out how to delete this partition and I don''t know why >>> it''s there. How do I mirror the root pool if I don''t have both disks >>> attached during OpenSolaris installation? I realize I can just use a >>> disk larger than 16GBs, but that would be a waste. >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Hi Frank, This bug was filed with bugster, but I see that the opensolaris bug database is currently unavailable. I sent a note about this problem. When a root cause is determined for 6844090, then we''ll see whether this particular issue is a ZFS problem or a format/fdisk problem. In any case, improvements in this area are being discussed. ZFS brings us light years ahead in terms of ease of use, but crappy disk format/label/slice stuff brings us back. I just want to use disks, I don''t want to know them intimately. Why do I have to? Cheers, Cindy Frank Middleton wrote:> On 06/04/09 06:44 PM, Cindy.Swearingen at Sun.COM wrote: > >> Hi Noz, >> >> This problem was reported recently and this bug was filed: >> >> 6844090 zfs should be able to mirror to a smaller disk > > > Is this filed on bugs or defects? I had the exact same problem, > and it turned out to be a rounding error in Solaris format/fdisk. > The only way I could fix it was to use Linux (well, Fedora) sfdisk > to make both partitions exactly the same number of bytes. The > alternates partition seems to be hard wired on older disks and > AFAIK there''s no way to use that space. sfdisk is on the Fedora > live CD if you don''t have a handy Linux system to get it from. > BTW the disks were nominally the same size but had different > geometries. > > Since I can''t find 6844090, I have no idea what it says, but this > really seems to be a bug in fdisk, not ZFS, although I would think > ZFS should be able to mirror to a disk that is only a tiny bit > smaller... > > -- Frank > > >> I believe slice 9 (alternates) is an older method for providing >> alternate disk blocks on x86 systems. Apparently, it can be removed by >> using the format -e command. I haven''t tried this though. >> >> I don''t think removing slice 9 will help though if these two disks >> are not identical, hence the bug. >> >> You can workaround this problem by attaching a slightly larger disk. >> >> Cindy >> >> >> noz wrote: >> >>> I''ve been playing around with zfs root pool mirroring and came across >>> some problems. >>> >>> I have no problems mirroring the root pool if I have both disks >>> attached during OpenSolaris installation (installer sees 2 disks). >>> >>> The problem occurs when I only have one disk attached to the system >>> during install. After OpenSolaris installation completes, I attach the >>> second disk and try to create a mirror but I cannot. >>> >>> Here are the steps I go through: >>> 1) install OpenSolaris onto 16GB disk >>> 2) after successful install, shutdown, and attach second disk (also >>> 16GB) >>> 3) fdisk -B >>> 4) partition >>> 5) zfs attach >>> >>> Step 5 fails, giving a "disk too small" error. >>> >>> What I noticed about the second disk is that it has a 9th partition >>> called "alternates" that takes up about 15MBs. This partition doesn''t >>> exist in the first disk and I believe is what''s causing the problem. I >>> can''t figure out how to delete this partition and I don''t know why >>> it''s there. How do I mirror the root pool if I don''t have both disks >>> attached during OpenSolaris installation? I realize I can just use a >>> disk larger than 16GBs, but that would be a waste. >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>