zfs discuss - Dec 2008 - Noob: Best way to replace a disk when you''re out of internal connectors?

Hope I''m posting this in the right place.

I''ve got a RAIDZ2 volume made of 14 SATA 1TB drives.  The box
they''re in is absolutely packed full; I know of no way to add any
additional drives, or internal SATA connectors.  I have a dying drive in the
array (hereafter "drive N").  Obviously I should replace it.  But how?

The best method would be to add the new drive, "zpool replace" drive N
with drive 15, and remove the old drive.  But I''d have to use an
external enclosure for drive 15, so I wouldn''t want this to be the
permanent layout.  I actually tried it this way, thinking that I could then
remove the old drive N, move "drive 15" into the chassis and give it
drive N''s connector, and go from there.  But I could not for the life
of me figure out how to tell ZFS "I moved drive 15 to drive N".

I could swap the dying drive with the fresh drive, then run an in-place
"zpool replace".  But the drive isn''t [i]dead,[/i] it is
merely [i]dying[/i].  That seems like overkill.

What''s the best approach to replacing drives if I''m out of
internal connectors?

Thanks!
-- 
This message posted from opensolaris.org

On Mon, Dec 29, 2008 at 8:45 PM, Larry Hastings <larryatos at
hastings.org> wrote:
> Hope I''m posting this in the right place.
>
> I''ve got a RAIDZ2 volume made of 14 SATA 1TB drives.  The box
they''re in is absolutely packed full; I know of no way to add any
additional drives, or internal SATA connectors.  I have a dying drive in the
array (hereafter "drive N").  Obviously I should replace it.  But how?
Use a USB enclosure for the new drive, and do:
zfs replace bad_disk new_disk

You should be able to export the volume and physically replace the
disk at that point.

-B

-- 
Brandon High : bhigh at freaks.com

On Mon, 29 Dec 2008, Larry Hastings wrote:
>
> I could swap the dying drive with the fresh drive, then run an 
> in-place "zpool replace".  But the drive isn''t
[i]dead,[/i] it is
> merely [i]dying[/i].  That seems like overkill.
I am a bit slow today.  It seems like a dying drive should be replaced 
ASAP.  Since you are using RAIDZ2, replacing the drive as described 
above should not be a problem.  Is the issue that your hardware does 
not support hot swap and this is not a good time to shut the system 
down?

Regarding the proposal to replace the drive with an external USB 
drive, this approach will surely "work" but since USB is only good for
about 11 or 12MB/second, performance of the whole raidz2 vdev would 
surely suffer and writes would then be limited by USB speeds.  It is 
likely to take quite a long time to resilver to the USB drive and if 
the filesystem is busy, maybe it will never catch up.  It may perform 
better with the dying drive.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Tue, Dec 30, 2008 12:18 AM, Brandon High wrote:
> You should be able to export the volume and
> physically replace the disk at that point.
Again, noob here, so just making sure I understand what you suggest:
1) zpool replace volume baddisk newdisk
2) zpool export volume
3) physically remove baddisk and replace it with newdisk (which for me requires
shutting down, sigh)
4) zpool import volume

Is that right?

Thanks,


/larry/
-- 
This message posted from opensolaris.org

> I am a bit slow today.  It seems like a dying drive should be replaced 
> ASAP. 
Completely agree with Bob on this.   I drive an 8.000lb truck and the
tires have industrial strength runflats.   If I get a puncture or tear
in a tire I replace it as soon as I can, not when it is convenient.
The runflats get me out of the woods or down the street.   Since you
are running RAIDZ2 then the better analogy might be half-shafts,
but you get the point.
>  Since you are using RAIDZ2, replacing the drive as described 
> above should not be a problem.  Is the issue that your hardware does 
> not support hot swap and this is not a good time to shut the system 
> down?
I would recommend a system maintenance window for failing hardware as
soon as you can reasonably do it.   You still have protection for your
data, but a flaky drive needs to be replaced.
> Regarding the proposal to replace the drive with an external USB 
> drive, this approach will surely "work" but since USB is only
good for
> about 11 or 12MB/second, performance of the whole raidz2 vdev would 
> surely suffer and writes would then be limited by USB speeds.  It is 
> likely to take quite a long time to resilver to the USB drive and if 
> the filesystem is busy, maybe it will never catch up.  It may perform 
> better with the dying drive.
Exactly.   The analogy here is the space saver spare tire.   use only
as a last resort :-)


Bob

On Tue, 30 Dec 2008, Larry Hastings wrote:
> Again, noob here, so just making sure I understand what you suggest:
> 1) zpool replace volume baddisk newdisk
> 2) zpool export volume
> 3) physically remove baddisk and replace it with newdisk (which for 
> me requires shutting down, sigh)
> 4) zpool import volume
>
> Is that right?
I don''t see a need for the zpool export and import unless you have a 
way to convert your USB drive into an drive which works directly in 
your chassis.  Once you have replaced the disk, you could do something 
like

   zpool replace mypool usbdisk replaceddisk

and wait another long while for the resilvering.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Tue, 30 Dec 2008, Bob Friesenhahn wrote:
> I don''t see a need for the zpool export and import
> unless you have a way to convert your USB drive
> into an drive which works directly in your chassis.
Ah, but that''s exactly what I''ve got.  I''m not using
a dedicated USB drive, I''m using a simple external USB drive harness (a
"Coolmax CD-350-Combo").  I can easily unplug the harness and move the
drive into my server.

So it sounds like the procedure I outlined above is the right way to go. 
Thanks, folks!  I''ve already started the (sixteen-hour) replace.


/larry/
-- 
This message posted from opensolaris.org

Umm, why do you need to do it the complicated way ? Here it is from zpool man
page-

     zpool replace [-f] pool old_device [new_device]

         Replaces old_device with new_device. This is  equivalent
         to attaching new_device, waiting for it to resilver, and
         then detaching old_device.

         The size of new_device must be greater than or equal  to
         the minimum size of all the devices in a mirror or raidz
         configuration.

         new_device is required if the pool is not redundant.  If
         new_device  is not specified, it defaults to old_device.
         This form of replacement is  useful  after  an  existing
         disk  has  failed  and has been physically replaced.  In
         this case, the new disk may have the same /dev/dsk  path
         as  the  old  device,  even though it is actually a dif-
         ferent disk. ZFS recognizes this.



You just need to hot-swap the failed disk (say c0d0) and utter "zpool
replace c0d0" and go have a coffee or something.
-- 
This message posted from opensolaris.org

On Tue, Dec 30, 2008 at 12:18 AM, Brandon High <bhigh at freaks.com>
wrote:
> Use a USB enclosure for the new drive, and do:
> zfs replace bad_disk new_disk
>
> You should be able to export the volume and physically replace the
> disk at that point.
It was late when I wrote that, so let me clarify a few things.

I''m assuming that this is not a busy production server. If it is, you
may want to do things differently. You said you can''t hot swap the
disk so you need to shutdown regardless. Plan on downtime.

Resilvering to the USB device will be slow. USB 2.0 is closer to 35-40
MB/s but 1TB is a lot of bits.

You don''t need to tell ZFS that the disk which had been in the USB
enclosure is now attached to a SATA bus. ZFS uses GUIDs on the
devices, not the bus path. It''ll just work. Hopefully.

On Tue, Dec 30, 2008 at 8:37 AM, Larry Hastings <larryatos at
hastings.org> wrote:
> Again, noob here, so just making sure I understand what you suggest:
0) Add the new disk to the system via USB, eSATA, firewire, whatever.
> 1) zpool replace volume baddisk newdisk
1.5) Wait for resilver.
1.6) Wait some more.
1.7) Continue waiting. You get the point.
> 2) zpool export volume
> 3) physically remove baddisk and replace it with newdisk (which for me
requires shutting down, sigh)
Shutdown & power off. Remove the failing drive. Remove the new drive
from the enclosure and install on the SATA port that the failed drive
was using.
> 4) zpool import volume
-B

-- 
Brandon High : bhigh at freaks.com

On Tue, Dec 30, 2008 at 2:58 PM, Brandon High <bhigh at freaks.com>
wrote:
> 4) zpool import volume
Alas--it did not work.

root at elephant# zpool import home
cannot import ''home'': no such pool available

I was able to import it by force:

root at elephant# zpool import -d /dev/mapper home

I exported it again, then ran "zpool import home" again and got the
same error. "zpool import -d" still works, so I still have the
partition.  But:
> You don''t need to tell ZFS that the disk which had been in the USB
> enclosure is now attached to a SATA bus. ZFS uses GUIDs on the
> devices, not the bus path. It''ll just work. Hopefully.
Alas--it did not work.  It still looks for the drive under the old name.

Any more suggestions?


/larry/
-- 
This message posted from opensolaris.org