zfs discuss - Apr 2010 - Create 1 pool from 3 exising pools in mirror configuration

[Note: This discussion started in another thread

    Subject: about backup and mirrored pools

but the subject has been significantly changed so started a new
thread]

Bob Friesenhahn <bfriesen at simple.dallas.tx.us> writes:
> Luckily, since you are using mirrors, you can easily migrate disks
> from your existing extra pools to the coalesced pool.  Just make sure
> to scrub first in order to have confidence that there won''t be
data
> loss.
Reading some of the info at the migration documentation:

 http://docs.huihoo.com/opensolaris/solaris-zfs-administration-guide/html/ch04s06.html

I''m getting a little (read horribly) confused how to go about doing
something like creating a single zpool of 3 sets of currently mirrored
disks that for each pair, constitute a zpool themselves.

So, I have rpool z2 z3. Each has 2 disks setup in mirror
configuration.

 zpool list
  NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
  rpool   466G   377G  88.7G    80%  1.00x  ONLINE  -
  z2      464G   365G  99.1G    78%  1.00x  ONLINE  -
  z3      696G   130G   566G    18%  1.00x  ONLINE  -

(`zpool status'' on all pools is inlined at the end of this post)

Whats confusing me is how to do this when all elements of the new
zpool are already in use as zpools themselves, especially since one of
them is rpool.

`export'' apparently at least tries to unmount the pool so taking them
out of service, but I guess whats bugging me is how to create a zpool
when the drives you plan to use are already zpools.

I''d like the existing mirrored sets to become the vdevs (I hope
that''s
the right term) making up the new zpool.  So not sure how to create
the new pool.

My only experience so far has been creating zpools from empty
hard drives I''d installed for that purpose.

Apparently some kind of usage of export/import will get this done.
But the docs at the above address are leaving me too unprepared to
attempt it without some first hand guidance from someone who has done
it.  Especially regarding rpool.

Can you create a zpool with no drives/vdevs and then `import'' each of
the
current pools into it, letting them become the vdevs?

When I export rpool it will be unmounted I guess, so does that have to
be done from a livecd?

In the case I''m describing I guess rpool will be the only pool when
its completed.  Is that a sensible thing to do?

Or would it make more sense to leave rpool alone and make a single
zpool of the other two mirrored pairs?

-------        ---------       ---=---       ---------      -------- 
  pool: rpool
 state: ONLINE
 scrub: scrub completed after 4h36m with 0 errors on Sat Apr 10 07:40:12 2010
config:

	NAME        STATE     READ WRITE CKSUM
	rpool       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    c3d0s0  ONLINE       0     0     0
	    c3d1s0  ONLINE       0     0     0

errors: No known data errors

  pool: z2
 state: ONLINE
 scrub: scrub completed after 3h44m with 0 errors on Fri Apr  9 06:48:10 2010
config:

	NAME        STATE     READ WRITE CKSUM
	z2          ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    c4d0    ONLINE       0     0     0
	    c4d1    ONLINE       0     0     0

errors: No known data errors

  pool: z3
 state: ONLINE
 scrub: scrub completed after 0h54m with 0 errors on Thu Apr  8 03:58:45 2010
config:

	NAME        STATE     READ WRITE CKSUM
	z3          ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    c5d0    ONLINE       0     0     0  60.5K repaired
	    c6d0    ONLINE       0     0     0

errors: No known data errors

On Sat, Apr 10, 2010 at 02:51:45PM -0500, Harry Putnam
wrote:
> [Note: This discussion started in another thread
> 
>     Subject: about backup and mirrored pools
> 
> but the subject has been significantly changed so started a new
> thread]
> 
> Bob Friesenhahn <bfriesen at simple.dallas.tx.us> writes:
> 
> > Luckily, since you are using mirrors, you can easily migrate disks
> > from your existing extra pools to the coalesced pool.  Just make sure
> > to scrub first in order to have confidence that there won''t
be data
> > loss.
You know, when I saw those words, I worried someone, somewhere would
interpret them incorrectly.

The migration he''s referring to is of disks, not of contents.  The
contents you''d have to migrate first (say with send|recv), before
destroying the emptied pool and adding the disks to the pool you want
to expand, as a new vdev.    There''s an implicit requirement here for
free space (or staging space elsewhere) to enable the move.

Note, also, rpool can''t have multiple vdevs, so the best you could
combine curently is z2 and z3.
> I''m getting a little (read horribly) confused how to go about
doing
> something like creating a single zpool of 3 sets of currently mirrored
> disks that for each pair, constitute a zpool themselves.
This would be something like a "zpool merge" operation, which does not
exist.
> In the case I''m describing I guess rpool will be the only pool
when
> its completed.  Is that a sensible thing to do?
No, as above.  You might consider new disks for a new rpool (say, ssd
with some zil or l2arc) and reusing the current disks for data if
they''re the same as the other data disks.
> Or would it make more sense to leave rpool alone and make a single
> zpool of the other two mirrored pairs?
Yep.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100411/b94e1e48/attachment.bin>

On Sun, 11 Apr 2010, Daniel Carosone wrote:
>
> The migration he''s referring to is of disks, not of contents.  The
> contents you''d have to migrate first (say with send|recv), before
> destroying the emptied pool and adding the disks to the pool you want
> to expand, as a new vdev.    There''s an implicit requirement here
for
> free space (or staging space elsewhere) to enable the move.
Since he is already using mirrors, he already has enough free space 
since he can move one disk from each mirror to the "main" pool (which 
unfortunately, can''t be the boot ''rpool'' pool), send
the data, and
then move the second disks from the pools which are be removed.  The 
main risk here is that there is only single redundancy for a while.
> No, as above.  You might consider new disks for a new rpool (say, ssd
> with some zil or l2arc) and reusing the current disks for data if
> they''re the same as the other data disks.
It is nice to have a tiny disk for the root pool.  An alternative is 
to create a small partition on two of the boot disks for use as the 
root pool, and use the remainder (''hog'') partition for the
main data
pool.  It is usually most convenient in the long run for the root pool 
to be physically separate from the data storage though.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Sat, 10 Apr 2010, Bob Friesenhahn wrote:
>
> Since he is already using mirrors, he already has enough free space since
he
> can move one disk from each mirror to the "main" pool (which
unfortunately,
> can''t be the boot ''rpool'' pool), send the data,
and then move the second
> disks from the pools which are be removed.  The main risk here is that
there
> is only single redundancy for a while.
I should mention that since this is a bit risky, it would be wise for 
Harry to post the procedure and commands he plans to use so that other 
eyes can verify that a correct result will be obtained.

There is more risk of configuring the pool incorrectly (e.g. failing 
to end up with mirror vdevs) than there is of losing data.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Sat, Apr 10, 2010 at 06:20:54PM -0500, Bob Friesenhahn
wrote:
> Since he is already using mirrors, he already has enough free space  
> since he can move one disk from each mirror to the "main" pool
(which
> unfortunately, can''t be the boot ''rpool'' pool),
send the data, and then
> move the second disks from the pools which are be removed. 
Ah, right you are. D''oh.
> The main risk here is that there is only single redundancy for a while.
You mean single copy, no redundancy, but otherwise yes.. perhaps
that''s why I hadn''t noticed this scheme, but it was a
subconscious
oversight.  I''d rather consider and eliminate it consciously if so. 

For Harry''s benefit, the recipe we''re talking about here is
roughly as
follows. Your pools z2 and z3, we will merge into z2.  <diskx> and
<disky> are the current members of z3.
 
Break the z3 mirror

# zpool detach z3 <diskx>

Add a new vdev to z2

# zpool add -f z2 <diskx>

The -f may be necessary, since you''re adding a vdev with different
redundancy profile to the existing vdev. 

Replicate the z3 data into z2

# zfs snapshot -R z3 at move
# zfs create z2/z3
# zfs send -R z3 at move | zfs recv -d z2/z3

Free up the second z3 disk and attach as a mirror

# zpool destroy z3
# zpool attach z2 <diskx> <disky>

Again, commands are approximate to illustrate the steps; in particular
you might choose a differnet replication structure.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100411/5d05cfd6/attachment.bin>

Daniel Carosone <dan at geek.com.au> writes:

Thanks for the input.. very helpful.

[...]
> No, as above.  You might consider new disks for a new rpool (say, ssd
> with some zil or l2arc) and reusing the current disks for data if
> they''re the same as the other data disks.
Would you mind expanding the abbrevs: ssd zil 12arc?

On Sat, 10 Apr 2010, Harry Putnam wrote:
>
> Would you mind expanding the abbrevs: ssd zil 12arc?
SSD   = Solid State Device
ZIL   = ZFS Intent Log (log of pending synchronous writes)
L2ARC = Level 2 Adaptive Replacement Cache

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On 04/11/10 11:55 AM, Harry Putnam wrote:
> Would you mind expanding the abbrevs: ssd zil 12arc?
>
>    
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

-- 
Ian.

Daniel Carosone <dan at geek.com.au> writes:
> For Harry''s benefit, the recipe we''re talking about here
is roughly as
> follows. Your pools z2 and z3, we will merge into z2.  <diskx> and
> <disky> are the current members of z3.
[...] snipped very handy outline

Thank you.  That kind of walk through is really helpful here.

I haven''t actually done any of it yet, still cogitating.

Also though, for my lightish usage... as home lan NAS I kind of wonder
if its worth trying that.  It would be nice to make better use of the
space I have.

But as someone suggested it might be better to get two more bigger
drives.  1t or 1.5t would handle all my data on one pair.

Then I guess after moving all the data to a single zpool made up of
those 2 new disks, I could then add the freed up drives as vdevs to
it?

So if you have time and inclination another walk through scheme would
be well appreciated.

I think, I''d just leave rpool clear out of it.  That mirror is made of
2 500gb IDE drives, that are now almost outdated by the move to sata.

rpool consists of 2 500gb IDE drives.
z2 consists of 2 500gb sata drives
z3 consists of 2 750 gb sata drives.

So I guess I''d add 2 1.5tb sata drives and create z1 of them.

I think I might know how to migrate all the data or at least I''m
confident with a little reading of opensolaris docs I will be able to
do it.

I''m not clear how I''d go about adding the freed up drives to
z1. ending
up with 3vdevs in z1, each a 2disk mirrors.

Or what commands then will stripe(?) the data across them all.

Once done, I would then have some thing close to 2.7TB available on
z1.

I''m wondering if after completing the above migrations, I''d
then need
to migrate whatever data is now being kept on rpool, onto the new z1
and just leave those 2 rpool disks for the OS, even though it would
waste a few hundred GB.  But maybe keeping the periodic hefty
delete/write off the system disks is worth it?

Or would there really be any good reason not to just leave some data
on rpool?  Maybe certain kinds of data that don''t grow to speak of.
Is it really that bad having some data stored on the OS (rpool)
drives?  I''m thinking maybe keeping system (C drive) disk images of
the windows lan machines there, which would mean periodic hefty
deletes/writes but the amount of data would not really grow too much.

On Mon, Apr 12, 2010 at 7:38 AM, Harry Putnam <reader at newsguy.com>
wrote:
> But as someone suggested it might be better to get two more bigger
> drives. ?1t or 1.5t would handle all my data on one pair.
>
> Then I guess after moving all the data to a single zpool made up of
> those 2 new disks, I could then add the freed up drives as vdevs to
> it?
It''s probably even easier than that if you have enough ports and bays
for two more drives. What you''d have to do, roughly, is:

1. Add both of the new drives as additional mirrors of the z2 pool.
Wait for resilver to complete.
2. Detach the original drives from z2. If autoexpand is set to on, z2
should now have 1.5TB. Otherwise export / import the pool.
3. Add the original drives to z2 as a mirror vdev.
4. Do a zfs send of all datasets from z3 to z2.
5. zpool destroy z3.
6. Add the drives from z3 to z2 a mirror vdev.

Your z2 should now have about 2.75TB space.

If you''re so inclined, you could move some datasets from rpool to z2
to keep your rpool smaller.

-B

-- 
Brandon High : bhigh at freaks.com

Brandon High <bhigh at freaks.com> writes:
> On Mon, Apr 12, 2010 at 7:38 AM, Harry Putnam <reader at newsguy.com>
wrote:
>> But as someone suggested it might be better to get two more bigger
>> drives. ?1t or 1.5t would handle all my data on one pair.
>>
>> Then I guess after moving all the data to a single zpool made up of
>> those 2 new disks, I could then add the freed up drives as vdevs to
>> it?
First off I''m sorry to keep on with what are actually theoretical
questions since I''m chicken to work along with this with real
experiments etc.... It may get a little irritating answering things that
might be obvious if I just tried this stuff as I go allong.  

But, I''m too unskilled in solaris and zfs admin to be risking a total
melt down if I try that before gaining a more thorough understanding.
> It''s probably even easier than that if you have enough ports and
bays
> for two more drives. What you''d have to do, roughly, is:
>
> 1. Add both of the new drives as additional mirrors of the z2 pool.
> Wait for resilver to complete.
> 2. Detach the original drives from z2. If autoexpand is set to on, z2
> should now have 1.5TB. Otherwise export / import the pool.
> 3. Add the original drives to z2 as a mirror vdev.
I''m getting a bit confused here.

If its possible to add a mirrored set as a vdev to a zpool like what
seems to be happening in (3) above, why wouldn''t I just add the two
new disks as mirrored vdev to z2 to start off, rather than additional
mirrors, and never remove the original disks of z2.
> 4. Do a zfs send of all datasets from z3 to z2.
> 5. zpool destroy z3.
> 6. Add the drives from z3 to z2 a mirror vdev.
[...]
> If you''re so inclined, you could move some datasets from rpool to
z2
> to keep your rpool smaller.
So having some data on rpool (besides the OS I mean) is not
necessarily a bad thing then?

On Mon, Apr 12, 2010 at 06:17:47PM -0500, Harry Putnam
wrote:
> But, I''m too unskilled in solaris and zfs admin to be risking a
total
> melt down if I try that before gaining a more thorough understanding.
Grab virtualbox or something similar and set yourself up a test
environment.  In general, and for you in particular, you will learn
the most this way - including learning what not to fear. 

You can also experiment with test pools in files, using a file per
vdev instead of a disk per vdev.   There''s no need for these vdevs to
be especially large, in order to practice things like attaching
mirrors or sending between pools.
> If its possible to add a mirrored set as a vdev to a zpool like what
> seems to be happening in (3) above, why wouldn''t I just add the
two
> new disks as mirrored vdev to z2 to start off, rather than additional
> mirrors, and never remove the original disks of z2.
If you have enough ports and bays for all the drives, sure.  There was
an assumption from your earlier messages that the 1.5''s were to
replace the original drives.  This involves the same consolidation
steps as before, with an add of the new 1.5TB mirror set as an
additional step, basically anywhere in the sequence.

Oh, another thing, just to make sure before you start, since this is
evidently older hardware: are you running a 32-bit or 64-bit kernel?
The 32-bit kernel won''t use drives larger than 1TB.
> So having some data on rpool (besides the OS I mean) is not
> necessarily a bad thing then?
Not at all; laptops would be screwed otherwise.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100413/507ca0cc/attachment.bin>

On Mon, Apr 12, 2010 at 4:17 PM, Harry Putnam <reader at newsguy.com>
wrote:
> If its possible to add a mirrored set as a vdev to a zpool like what
> seems to be happening in (3) above, why wouldn''t I just add the
two
> new disks as mirrored vdev to z2 to start off, rather than additional
> mirrors, and never remove the original disks of z2.
I started writing it, assuming that you''d discard the original two
drives, then realized you''d said you were going to add the new drives
and keep the old but I didn''t clean things up completely.

What that, the process is:
1. Add the new drives as a mirror vdev to the z2 pool. ( zpool attach
z2 mirror new_disk_0 new_disk_0 - The "mirror is very, very important.
Without it you''ll add two unmirrored vdevs. zpool should complain if
this is the case though.)
2. Do a zfs send of all datasets from z3 to z2.
3. zpool destroy z3.
4. Add the drives from z3 to z2 as a mirror vdev.

As Daniel suggested, testing the process on VirtualBox / VMWare / etc
is a great idea.
>> If you''re so inclined, you could move some datasets from rpool
to z2
>> to keep your rpool smaller.
>
> So having some data on rpool (besides the OS I mean) is not
> necessarily a bad thing then?
The pool will resilver faster if it''s got less data on it, which may
be important to you.

The rpool only supports mirrored redundancy, and you can''t add more
vdevs to it, so the ability to grow it is limited.

It''s also good practice to keep your OS install separate from your
data for maintenance reasons. For my home server, the rpool contains
only the OS install and everything else is in a separate pool.

-B

-- 
Brandon High : bhigh at freaks.com

Daniel Carosone <dan at geek.com.au> writes:

[...] snipped welcome info... thanks.
> Oh, another thing, just to make sure before you start, since this is
> evidently older hardware: are you running a 32-bit or 64-bit kernel?
> The 32-bit kernel won''t use drives larger than 1TB.
Its an athlon64 so I''m good there.

Brandon High <bhigh at freaks.com> writes:

[...]
Harry wrote:
>> So having some data on rpool (besides the OS I mean) is not
>> necessarily a bad thing then?
Daniel C answered:
>> Not at all; laptops would be screwed otherwise. 
Brandon H. responded:
> The pool will resilver faster if it''s got less data on it, which
may
> be important to you.
> 
> The rpool only supports mirrored redundancy, and you can''t add
more
> vdevs to it, so the ability to grow it is limited.
> 
> It''s also good practice to keep your OS install separate from your
> data for maintenance reasons. For my home server, the rpool contains
> only the OS install and everything else is in a separate pool.
Apparently you are not disagreeing with Daniel Cs'' comment above so I
guess you are talking about disk partitions here?  

Currently two 500gb IDE drives make up rpool.  So to have mirror
redundancy of rpool, do you mean to create identical partitions on
each disk for mirrored rpool, then use the other partition for data?

I thought I remembered being told it was better not to partition disks
but I''m probably just confused about it.

On Tue, Apr 13, 2010 at 7:03 AM, Harry Putnam <reader at newsguy.com>
wrote:
> Apparently you are not disagreeing with Daniel Cs'' comment above
so I
> guess you are talking about disk partitions here?
I''m not disagreeing, but the use case for a server is different than
for a laptop that can only hold one drive.
> Currently two 500gb IDE drives make up rpool. ?So to have mirror
> redundancy of rpool, do you mean to create identical partitions on
> each disk for mirrored rpool, then use the other partition for data?
Er, not exactly.

There are a few constraints on rpools and on zpools that limit what
you can do for the rpool.

The first constraint is that it only supports one top-level vdev, and
that vdev can only be non-redundant or a mirror. If the pool fills up,
you can''t grow it by adding more disks. This limits your future
growth. You could replace the drives with larger ones, but that''s
about it. Once it fills up, you have to start deleting.

By default the write cache is disabled when zfs doesn''t have access to
the whole disk, and due to the way BIOS boots, the boot drive is
partitioned. The rpool will have slightly worse performance as a
result. Partitioning out a smaller boot volume and using the rest in a
second zpool will just wind up with the second pool having poor
performance too. You could add more vdevs to the second pool (or add
them one that already exists) but performance will also be unbalanced
since the partitions will be slower. It''s not a pretty picture.
It''ll
work, it just won''t offer the performance that you probably want.
> I thought I remembered being told it was better not to partition disks
> but I''m probably just confused about it.
You''re right, it''s best not to. If you want to keep your rpool
separate, dedicate a drive (or two) as the boot drives and leave it at
that. Move any user data to your larger (and better performing) pools
and call it a day.

-B

-- 
Brandon High : bhigh at freaks.com