zfs discuss - Feb 2010 - Adding a zfs mirror drive to rpool - new drive formats to one cylinder less

Hi guys,
   I have just installed open solaris 2009.6 on my server using a 250G laptop
drive (using the entire drive).

I now want to add another [u]identical[/u] 250G laptop drive to the rpool to
create a mirror. The have been no other changes to hardware (x86 -  intel entry
level server board).

All the instructions I have seen indicate i should format the drive with a 100%
solaris partition...eg here
http://darkstar-solaris.blogspot.com/2008/09/zfs-root-mirror.html

The problem is, when I format the drive using "format",
"fdisk" and 100% solaris - yes and then look at the drive size with
"format", I see that t[b]he "new" drive is one cylinder less
than the current rpool drive[/b] (C8t0d0).

eg existing rpool drive c8t0d0 = 39999 cyl
 newly formatted drive c8t1d0 = 3999[b]8[/b] cyl
(not actual numbers as I am not infront of the server at the moment - just an
example)

So, naturally when I run ...prtvtoc /dev/rdsk/c8t1d0s2 | fmthard -s -
/dev/rdsk/c8t1d0S2, it complains about the size of the disk.

What am I doing wrong? How can I get the second drive c8t1d0 to format to
exactly the same size as the original?

I am a newbie to opensolaris so please provide step by step instructions as I
have tried to find the answer myself all day but can not.  The closest i have
come is a statement that I need to manually set the partition size...but how?

I also have an 250G desktop drive that I tested and it behaves exactly the same
(ie. formats to one less cylinder).

Thanks.
-- 
This message posted from opensolaris.org

On Mon, Feb 22, 2010 at 22:55, tomwaters <tomwaters at chadmail.com>
wrote:
> Hi guys,
>   I have just installed open solaris 2009.6 on my server using a 250G
> laptop drive (using the entire drive).
>
> I now want to add another [u]identical[/u] 250G laptop drive to the rpool
> to create a mirror. The have been no other changes to hardware (x86 - 
intel
> entry level server board).
>
> All the instructions I have seen indicate i should format the drive with a
> 100% solaris partition...eg here
> http://darkstar-solaris.blogspot.com/2008/09/zfs-root-mirror.html
>
> The problem is, when I format the drive using "format",
"fdisk" and 100%
> solaris - yes and then look at the drive size with "format", I
see that
> t[b]he "new" drive is one cylinder less than the current rpool
drive[/b]
> (C8t0d0).
>
> eg existing rpool drive c8t0d0 = 39999 cyl
>  newly formatted drive c8t1d0 = 3999[b]8[/b] cyl
> (not actual numbers as I am not infront of the server at the moment - just
> an example)
>
> So, naturally when I run ...prtvtoc /dev/rdsk/c8t1d0s2 | fmthard -s -
> /dev/rdsk/c8t1d0S2, it complains about the size of the disk.
>
> What am I doing wrong? How can I get the second drive c8t1d0 to format to
> exactly the same size as the original?
>
> I am a newbie to opensolaris so please provide step by step instructions as
> I have tried to find the answer myself all day but can not.  The closest i
> have come is a statement that I need to manually set the partition
> size...but how?
>
> I also have an 250G desktop drive that I tested and it behaves exactly the
> same (ie. formats to one less cylinder).
>
> Thanks.
> --
>
>
Have you tried simply adding the c?t?d? device as a mirror to the zpool,
without bothering with formatting/partition tables/all that? My
understanding is that zfs defaults to adding a EFI partition table when
given raw drives (not vtoc, which it appears you are using there), and if
you just give the raw drive to the zpool, you shouldn''t need to deal
with
the format utility. (I''m rather new to solaris as well, I trust
somebody
will correct me if I''m mistaken about any of this.)

-Ethan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100222/6fb7ce63/attachment.html>

I don''t know if this will help or not, or if it has anything to do with
your
situation, but a guy who was having a problem on the list awhile back had
identical drives showing up as different sizes, it turned out to be that
they were both sata drives but some were on sata controllers, and the others
were on sata controllers on the motherboard but stuck in IDE mode.

This MAY be the issue.

On Mon, Feb 22, 2010 at 10:55 PM, tomwaters <tomwaters at chadmail.com>
wrote:
> Hi guys,
>   I have just installed open solaris 2009.6 on my server using a 250G
> laptop drive (using the entire drive).
>
> I now want to add another [u]identical[/u] 250G laptop drive to the rpool
> to create a mirror. The have been no other changes to hardware (x86 - 
intel
> entry level server board).
>
> All the instructions I have seen indicate i should format the drive with a
> 100% solaris partition...eg here
> http://darkstar-solaris.blogspot.com/2008/09/zfs-root-mirror.html
>
> The problem is, when I format the drive using "format",
"fdisk" and 100%
> solaris - yes and then look at the drive size with "format", I
see that
> t[b]he "new" drive is one cylinder less than the current rpool
drive[/b]
> (C8t0d0).
>
> eg existing rpool drive c8t0d0 = 39999 cyl
>  newly formatted drive c8t1d0 = 3999[b]8[/b] cyl
> (not actual numbers as I am not infront of the server at the moment - just
> an example)
>
> So, naturally when I run ...prtvtoc /dev/rdsk/c8t1d0s2 | fmthard -s -
> /dev/rdsk/c8t1d0S2, it complains about the size of the disk.
>
> What am I doing wrong? How can I get the second drive c8t1d0 to format to
> exactly the same size as the original?
>
> I am a newbie to opensolaris so please provide step by step instructions as
> I have tried to find the answer myself all day but can not.  The closest i
> have come is a statement that I need to manually set the partition
> size...but how?
>
> I also have an 250G desktop drive that I tested and it behaves exactly the
> same (ie. formats to one less cylinder).
>
> Thanks.
> --
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100223/c31516f2/attachment.html>

Thanks for the replies.

1) Both are on the same controller (Intel motherboard) - one on port 0, the
other in port 1 (I think is is demonstrated by c8t0d0 and c8t1d0 ?)

2) I have tried adding them without doing the format and no joy.

Any other ideas?

Message was edited by: tomwater
-- 
This message posted from opensolaris.org

Looks like an issue with the start /length of the partition table...

These are the disks from "fomrat"...
       8. c8t0d0 <DEFAULT cyl 30399 alt 2 hd 255 sec 63>
          /pci at 0,0/pci8086,34d0 at 1f,2/disk at 0,0
       9. c8t1d0 <DEFAULT cyl 30398 alt 2 hd 255 sec 63>
          /pci at 0,0/pci8086,34d0 at 1f,2/disk at 1,0

Loking at the partitions, the existing rpool disk is formatted like this..

             Total disk size is 30401 cylinders
             Cylinder size is 16065 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ==          1  
Active    Solaris2          0  30401    30402    100


The "new" disk is formatted like this...
             Total disk size is 30401 cylinders
             Cylinder size is 16065 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ==          1  
Active    Solaris2          1  30400    30400    100


I tried to manually create a partition by selecting "part" from the
menu but I can not tell it to start at cyl. 0 or go to 30399...

Help!

Why is this so hard!
-- 
This message posted from opensolaris.org

On Mon, 22 Feb 2010, tomwaters wrote:
> I have just installed open solaris 2009.6 on my server using a 250G 
> laptop drive (using the entire drive).
So, 2009.06 was based on 111b.  There was a fix that went into build 117 
that allows you to mirror to smaller disks if the metaslabs in zfs are 
still the same size even if the disk is one block smaller.  CR 6844090.

If you can upgrade to a later build, this will very probably fix your 
issue.


Regards,
markm

Thanks for that.

It seems strange though that the two disks, which are from the same
manufacturer, same model, same firmware and similar batch/serial''s
behave differently.

I am also puzzled that the rpool disk appears to start at cylinder 0 and not 1.

I did find this quote after googling for the CR6844090 at the best practices
page http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

"The size of the replacements vdev, measured by usable sectors, must be the
same or greater than the vdev being replaced. This can be confusing when whole
disks are used because different models of disks may provide a different number
of usable sectors. For example, if a pool was created with a "500
GByte" drive and you need to replace it with another "500 GByte"
drive, then you may not be able to do so if the drives are not of the same make,
model, and firmware revision. Consider planning ahead and reserving some space
by creating a slice which is smaller than the whole disk instead of the whole
disk. In Nevada, build 117, it might be possible to replace or attach a disk
that is slight smaller than the other disks in a pool. This is CR 6844090.
"
-- 
This message posted from opensolaris.org

On Tue, February 23, 2010 17:58, tomwaters wrote:
> Thanks for that.
>
> It seems strange though that the two disks, which are from the same
> manufacturer, same model, same firmware and similar batch/serial''s
behave
> differently.
I''ve found that the ways of writing labels and partitions in Solaris
are
arcane and unpredictable.  The two sides of my rpool mirror are slightly
different sizes, even though they''re identical drives installed at the
same time.  I presume it''s somehow something I did in the process, but
I
haven''t nailed down exactly what (and eventually got past the point of
caring).

I was replacing rpool disks, so the pattern was attach new, resilver,
attach new, resilver, remove old, remove old -- so it just auto-expanded
to the smaller of the two new at the time the second old was removed; I
didn''t run into the problem of being unable to attach the disk because
it''s too small.  (Also have to run installgrub of course, not listed in
above pattern.)
> I am also puzzled that the rpool disk appears to start at cylinder 0 and
> not 1.
Historical habit, I think, from when cylinders were much smaller and
filesystems expected boot information to be outside the filesystem space
(my understanding is that ZFS is set up to not overwrite where the boot
stuff would be in the slice ZFS is using, just in case it''s there).

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

Thanks David.

Re. the starting cylinder, it was more that one c8t0d0 the partition started at
zero and c8t1d0 it started at 1.

ie.
c8t0d0
Partition Status Type Start End Length %
========= ====== ============ ===== === ====== ==1 Active Solaris2 0 30401 30402
100

c8t1d0:
Partition Status Type Start End Length %
========= ====== ============ ===== === ====== ==1 Active Solaris2 1 30400 30400
100

All good now and yes, i did run installgrub...yet to test booting from c8t1d0
(with c8t0d0 removed).
-- 
This message posted from opensolaris.org

Just an update on this, I was seeing high CPU utilisation (100% on all 4 cores)
for ~10 seconds every 20 seconds when transfering files to the server using
Samba under 133.

So I rebooted and selected 111b and I no longer have the issue. Interestingly,
the rpool is still in place..as it should be.  So I have now set this 111b as my
default BE ...and removed /dev from the update package list using ...
$pfexec pkg set-publisher -O  http://pkg.opensolaris.org/release opensolaris.org

I am not sure if I had to run the installgrub again, but I did anyway.
#installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c8t1d0s0

So all looking good now.  I have my rpool mirror and am back on the
"stable" release version.

I am beginning to like opensolaris.
-- 
This message posted from opensolaris.org

On Thu, February 25, 2010 08:25, tomwaters wrote:
> So I rebooted and selected 111b and I no longer have the issue.
> Interestingly, the rpool is still in place..as it should be.  So I have
> now set this 111b as my default BE ...and removed /dev from the update
> package list using ...
> $pfexec pkg set-publisher -O  http://pkg.opensolaris.org/release
> opensolaris.org
And thanks for the report on the successful downgrade.  I''ve been
considering whether I should upgrade to dev to get a particular bug-fix,
or just wait for the next stable release,and slight worry that it might be
hard to fall back (I do understand it''s supposed to work; supposed to
be
just as easy as it turned out to actually be for you).

I''ll probably still just wait, at this point, since it''s so
close
(supposed to be March, these things don''t seem to slip much more than a
month usually), but it''s nice to have more info about one of the bits I
was worried about.
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info