zfs discuss - Sep 2008 - create raidz with 1 disk offline

Hi,

I''m about to convert my 3 * 1 TB raidz to a 5 * 1 TB raidz. Since raidz
can''t be grown like that I have to find some place to move the data to
temporarily while I reformat the raidz. However, I''m short on disk
space (which is why I''m adding 2 new disks).

So, is it possible to create a 5 * 1 TB raidz with 4 disks (i.e., with
one disk offline)? In that case I could use one of the 1 TB disks as
temporary storage. (I would first move 1 TB of data from the old raidz
to one of the new disks, then delete the old raidz and create the new
5-way raidz with 4 disks, then copy the 1 TB of data from the 5th disk
to the new raidz and finally add the 5th disk to the raidz.)

Cheers,
Marcus

On Sat, Sep 27, 2008 at 4:02 PM, Marcus Sundman <sundman at iki.fi>
wrote:
> So, is it possible to create a 5 * 1 TB raidz with 4 disks (i.e., with
> one disk offline)? In that case I could use one of the 1 TB disks as
The official answer is no, but it might be possible to fool Solaris
into letting you do it. I haven''t tired this, so I don''t know
if it''ll
work.

1. Create a sparse file on an existing filesystem. The size should be
the same as your disks.
2. Create the raidz with 4 of the drives and the sparse file.
3. Export the zpool.
4. Delete the sparse file
5. Import the zpool. It should come up as degraded, since one of its
vdevs is missing.
6. Copy your files onto the zpool.
7. replace the file vdev with the 5th disk.

Like I said, I haven''t tried this but it might work. I''d love
to hear
if it does.

-B

-- 
Brandon High : bhigh at freaks.com

"Brandon High" <bhigh at freaks.com>
wrote:
> On Sat, Sep 27, 2008 at 4:02 PM, Marcus Sundman <sundman at iki.fi>
> wrote:
> > So, is it possible to create a 5 * 1 TB raidz with 4 disks (i.e.,
> > with one disk offline)?
> 
> [...]
> 1. Create a sparse file on an existing filesystem. The size should be
> the same as your disks.
> 2. Create the raidz with 4 of the drives and the sparse file.
> [...]
That''s brilliant in its simplicity, and I see no reason why it
wouldn''t
work. (ZFS even allows a larger sparse file than the size of the file
system it''s on.) I''m still waiting for my new disks and
SATA-card, but
I''ll send a follow-up message once I''ve got them and tried
your trick.
Thanks!

Regards,
Marcus

On 27 September, 2008 - Brandon High sent me these 1,0K bytes:
> On Sat, Sep 27, 2008 at 4:02 PM, Marcus Sundman <sundman at iki.fi>
wrote:
> > So, is it possible to create a 5 * 1 TB raidz with 4 disks (i.e., with
> > one disk offline)? In that case I could use one of the 1 TB disks as
> 
> The official answer is no, but it might be possible to fool Solaris
> into letting you do it. I haven''t tired this, so I don''t
know if it''ll
> work.
> 
> 1. Create a sparse file on an existing filesystem. The size should be
> the same as your disks.
> 2. Create the raidz with 4 of the drives and the sparse file.
> 3. Export the zpool.
> 4. Delete the sparse file
> 5. Import the zpool. It should come up as degraded, since one of its
> vdevs is missing.
> 6. Copy your files onto the zpool.
> 7. replace the file vdev with the 5th disk.
> 
> Like I said, I haven''t tried this but it might work. I''d
love to hear
> if it does.
I just tried (mkfile in /tmp) on both Sol10u5 and snv97, both seems to
work.

/Tomas
-- 
Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Ume?
`- Sysadmin at {cs,acc}.umu.se

Tomas ?gren wrote:
> On 27 September, 2008 - Brandon High sent me these 1,0K bytes:
>
>   
>> On Sat, Sep 27, 2008 at 4:02 PM, Marcus Sundman <sundman at
iki.fi> wrote:
>>     
>>> So, is it possible to create a 5 * 1 TB raidz with 4 disks (i.e.,
with
>>> one disk offline)? In that case I could use one of the 1 TB disks
as
>>>       
>> The official answer is no, but it might be possible to fool Solaris
>> into letting you do it. I haven''t tired this, so I
don''t know if it''ll
>> work.
>>
>> 1. Create a sparse file on an existing filesystem. The size should be
>> the same as your disks.
>> 2. Create the raidz with 4 of the drives and the sparse file.
>> 3. Export the zpool.
>> 4. Delete the sparse file
>> 5. Import the zpool. It should come up as degraded, since one of its
>> vdevs is missing.
>> 6. Copy your files onto the zpool.
>> 7. replace the file vdev with the 5th disk.
>>
>> Like I said, I haven''t tried this but it might work.
I''d love to hear
>> if it does.
>>     
>
> I just tried (mkfile in /tmp) on both Sol10u5 and snv97, both seems to
> work.
>   
Or use a ramdisk.  See ramdiskadm(1m)
[most people don''t seem to know Solaris has ramdisk devices]
 -- richard

Brandon High schrieb:
> 1. Create a sparse file on an existing filesystem. The size should be
> the same as your disks.
> 2. Create the raidz with 4 of the drives and the sparse file.
> 3. Export the zpool.
> 4. Delete the sparse file
> 5. Import the zpool. It should come up as degraded, since one of its
> vdevs is missing.
> 6. Copy your files onto the zpool.
> 7. replace the file vdev with the 5th disk.
You can skip steps 3-5 and just put the sparse file offline manually 
(zpool offline ...)

I have migrated a 2x1TB mirror to a 4x1TB RAIDZ this way a few weeks ago.


Daniel

> [most people don''t seem to know Solaris has ramdisk devices]
That is because only a select few are able to unravel the enigma wrapped in a
clue that is solaris :)
--
This message posted from opensolaris.org

> > [most people don''t seem to know Solaris has ramdisk devices]
>
> That is because only a select few are able to unravel the enigma wrapped in
a clue that is solaris :)
Hmmm... very enigmatic, your remark. :-)


However, in this case I suspect it is because ramdisks don''t really
work well on Solaris:

<shelob:/home/volker,2846# uname -a
SunOS shelob 5.10 Generic_137111-02 sun4v sparc SUNW,Sun-Fire-T1000
<shelob:/home/volker,2847# vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr lf lf lf s0   in   sy   cs us sy id
 0 0 0 33849968 2223440 2 14 1  0  0  0  0  0 21  0 21  813 1263  957  0  0 99
<shelob:/home/volker,2848# ramdiskadm
<shelob:/home/volker,2849# ramdiskadm -a testdisk 1g
ramdiskadm: couldn''t create ramdisk "testdisk": Resource
temporarily unavailable
Exit 1
<shelob:/home/volker,2850# ramdiskadm -a testdisk 768m
/dev/ramdisk/testdisk
<shelob:/home/volker,2851# vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr lf lf lf rm   in   sy   cs us sy id
 0 0 0 33849960 2223504 2 14 1  0  0  0  0  0 21  0 -0  813 1263  957  0  0 99
<shelob:/home/volker,2852# ramdiskadm -a testdisk2 768m
ramdiskadm: couldn''t create ramdisk "testdisk2": Resource
temporarily unavailable
Exit 1
<shelob:/home/volker,2853# vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr lf lf lf rm   in   sy   cs us sy id
 0 0 0 33849952 2223552 2 14 1  0  0  0  0  0 21  0 -0  813 1263  957  0  0 99


So they only work on and off.  I never bothered to find out what the
problem was (in fact, I hadn''t even tried the ramdiskadm cmd in that
version of Solaris before this email thread showed up).


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 45
Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt

On Mon, Sep 29, 2008 at 2:12 AM, Volker A. Brandt <vab at bb-c.de>
wrote:
> <shelob:/home/volker,2847# vmstat
>  kthr      memory            page            disk          faults      cpu
>  r b w   swap  free  re  mf pi po fr de sr lf lf lf s0   in   sy   cs us sy
id
>  0 0 0 33849968 2223440 2 14 1  0  0  0  0  0 21  0 21  813 1263  957  0  0
99
Note this from vmstat(1M):

     Without options, vmstat displays a one-line summary  of  the
     virtual memory activity since the system was booted.

In other words, the first line of vmstat output is some value that
does not represent the current state of the system.  Try this instead:

$ vmstat 1 2
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 m1   in   sy   cs us sy id
 0 0 0 12371648 5840200 30 93 14 1 1  0  0  0  1  0  5  711 2258 1257  1  0 98
 0 0 0 9581336 2987584 61 69 0  0  0  0  0  0  0  0  0  543  972  518  0  0 100
>From a free memory standpoint, the current state of the system is very
different than the typical state since boot.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

> Note this from vmstat(1M):
>
>      Without options, vmstat displays a one-line summary  of  the
>      virtual memory activity since the system was booted.
Oops, you''re correct.  I was only trying to demonstrate that there
was ample free memory and ramdiskadm just didn''t work.  Usually I do
that using top, but pasting top output into an email doesn''t really
parse very well.
> In other words, the first line of vmstat output is some value that
> does not represent the current state of the system.  Try this instead:
Yes, that gives correct numbers.  However, it would only lead away
from the problem with ramdiskadm(1M).  As such, the numbers are
almost identical on my system anyway.  :-)
> >From a free memory standpoint, the current state of the system is very
> different than the typical state since boot.
Quite right.


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 45
Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt

Volker A. Brandt wrote:
>>> [most people don''t seem to know Solaris has ramdisk
devices]
>>>       
>> That is because only a select few are able to unravel the enigma
wrapped in a clue that is solaris :)
>>     
>
> Hmmm... very enigmatic, your remark. :-)
>
>
> However, in this case I suspect it is because ramdisks don''t
really
> work well on Solaris:
>
> <shelob:/home/volker,2846# uname -a
> SunOS shelob 5.10 Generic_137111-02 sun4v sparc SUNW,Sun-Fire-T1000
> <shelob:/home/volker,2847# vmstat
>  kthr      memory            page            disk          faults      cpu
>  r b w   swap  free  re  mf pi po fr de sr lf lf lf s0   in   sy   cs us sy
id
>  0 0 0 33849968 2223440 2 14 1  0  0  0  0  0 21  0 21  813 1263  957  0  0
99
> <shelob:/home/volker,2848# ramdiskadm
> <shelob:/home/volker,2849# ramdiskadm -a testdisk 1g
> ramdiskadm: couldn''t create ramdisk "testdisk": Resource
temporarily unavailable
> Exit 1
> <shelob:/home/volker,2850# ramdiskadm -a testdisk 768m
> /dev/ramdisk/testdisk
> <shelob:/home/volker,2851# vmstat
>  kthr      memory            page            disk          faults      cpu
>  r b w   swap  free  re  mf pi po fr de sr lf lf lf rm   in   sy   cs us sy
id
>  0 0 0 33849960 2223504 2 14 1  0  0  0  0  0 21  0 -0  813 1263  957  0  0
99
> <shelob:/home/volker,2852# ramdiskadm -a testdisk2 768m
> ramdiskadm: couldn''t create ramdisk "testdisk2":
Resource temporarily unavailable
> Exit 1
> <shelob:/home/volker,2853# vmstat
>  kthr      memory            page            disk          faults      cpu
>  r b w   swap  free  re  mf pi po fr de sr lf lf lf rm   in   sy   cs us sy
id
>  0 0 0 33849952 2223552 2 14 1  0  0  0  0  0 21  0 -0  813 1263  957  0  0
99
>
>
> So they only work on and off.  I never bothered to find out what the
> problem was (in fact, I hadn''t even tried the ramdiskadm cmd in
that
> version of Solaris before this email thread showed up).
>   
AIUI, the memory assigned to a ramdisk must be contiguous.
This makes some sense in that they are designed to be bootable.
If you''ve been running for a while, the chance of finding large,
contiguous areas decreases.

With tmpfs and lofiadm, the RAM requirement does not exist,
but you are also not guaranteed to be using RAM.
 -- richard

> > So they only work on and off.  I never bothered to find out what the
> > problem was (in fact, I hadn''t even tried the ramdiskadm cmd
in that
> > version of Solaris before this email thread showed up).
> >
>
> AIUI, the memory assigned to a ramdisk must be contiguous.
> This makes some sense in that they are designed to be bootable.
> If you''ve been running for a while, the chance of finding large,
> contiguous areas decreases.
This is very interesting, thanks for enlightening me!  Maybe a small
note about this behavior is warranted in the NOTES section of
ramdiskadm(1M)? :-)


Thanks -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 45
Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt

Volker A. Brandt wrote:
>>> So they only work on and off.  I never bothered to find out what
the
>>> problem was (in fact, I hadn''t even tried the ramdiskadm
cmd in that
>>> version of Solaris before this email thread showed up).
>>>
>> AIUI, the memory assigned to a ramdisk must be contiguous.
>> This makes some sense in that they are designed to be bootable.
>> If you''ve been running for a while, the chance of finding
large,
>> contiguous areas decreases.
> 
> This is very interesting, thanks for enlightening me!  Maybe a small
> note about this behavior is warranted in the NOTES section of
> ramdiskadm(1M)? :-)
Probably ramdisk(7D) actually where it mentions the other memory 
limitations.

-- 
Darren J Moffat