zfs discuss - Sep 2006 - ZFS uses 1.1GB more space, reports conflicting information...

[Solaris 10 6/06 i86pc]

I recently used a set of 6 disks in a MultiPack to create a RAIDZ volume.  Then
I proceeded to do zfs set sharenfs=root=a.b.c.d:a.b.c.e space ("space"
is how I named the ZFS pool).

Then I NFS mounted the ZFS pool on another system, and proceeded to do a find +
cpio -pvdmu combination to the NFS mounted file system.

Shortly thereafter, I ran out of space on my "space" pool, but `zfs
list` kept reporting I still had about a GigaByte worth of free space, while
`zpool status` seemed to correctly report I ran out of space.

Why do these utilities report inconsistent information?

Then I added two more disks to the pool with the `zpool add -fn space c2t10d0
c2t11d0`, whereby I determined that those would be added as a RAID0, which is
not what I wanted. `zpool add -f raidz c2t10d0 c2t11d0` added ANOTHER RAIDZ
STRIPE to the pool, rather than adding the disks to the existing RAIDZ vdev!

Is there a way to add more disks to an existing RAIDZ vdev? If there is, I sure
haven''t figured it out yet. What is it?

And from what I understood from the blogs written by Mr. Bonwick, one of the
properties of RAIDZ is that it can be used on disks in a pool that are not the
same size. Why then did I explicitly have to use the -f switch to force a
creation of a RAIDZ vdev on a salad of 2 and 4GB disks?

Finally, after the find + cpio were done, I noticed that the ZFS file system is
taking exactly 1.1GB more space than the original UFS file system that the data
had been copied from. At least that''s so according to `du -sh`. Why???
 
 
This message posted from opensolaris.org

On 9/4/06, UNIX admin <tripivceta at hotmail.com>
wrote:
> [Solaris 10 6/06 i86pc]
>
> I recently used a set of 6 disks in a MultiPack to create a RAIDZ volume. 
Then I proceeded to do zfs set sharenfs=root=a.b.c.d:a.b.c.e space
("space" is how I named the ZFS pool).
Is this really how you set the sharenfs option?  If so, this will
default to exporting it read-write to everyone, even though the "rw"
does not show up in the share command output.

To get the behavior you probably want, you should use

    zfs set sharenfs=root=a.b.c.d:a.b.c.e,rw=a.b.c.d:a.b.c.e space

instead.  You must also specify a "ro=" or "rw=", as the
"root="
option is only used for mapping.  With the "root=access_list" syntax
and the wording of the share_nfs manual page this is easily confused.
I have reported these issues to the nfs-discuss@ list, and they will
be filed as bugs.

Chris

> Is this really how you set the sharenfs option?  If
It would certainly appear so. It took me a while to figure out the syntax
though; the manpage is ambigous on that.

It would be excellent if the manpage had more EXAMPLES. And if the EXAMPLES
section was actually named that instead of "Examples" (to be
consistent with other man pages, the unwritten convention and other system V
UNIXes as well).
> so, this will
> default to exporting it read-write to everyone, even
> though the "rw"
> does not show up in the share command output.
Yes, I am aware of that; but this was almost a one-time shot, where only I have
access to both systems which are in the inner sanctum, so it mattered but
little.
 
 
This message posted from opensolaris.org

Hi,

On 9/4/06, UNIX admin <tripivceta at hotmail.com>
wrote:
> [Solaris 10 6/06 i86pc]
...
> Then I added two more disks to the pool with the `zpool add -fn space
c2t10d0 c2t11d0`, whereby I determined that those would be added as a RAID0,
which is not what I wanted. `zpool add -f raidz c2t10d0 c2t11d0` added ANOTHER
RAIDZ STRIPE to the pool, rather than adding the disks to the existing RAIDZ
vdev!
That is correct.  The semantics of "zpool add" adds one or more
"vdev"s.  "raidz c2t10d0 c2t11d0" defines a new
"vdev" which is a
raidz of the 2 specified disks.
> Is there a way to add more disks to an existing RAIDZ vdev? If there is, I
sure haven''t figured it out yet. What is it?
AFAIK, no.  The "attach" semantics only works for adding mirrors.
Would be nice if that can be overloaded for RAIDZ.
> And from what I understood from the blogs written by Mr. Bonwick, one of
the properties of RAIDZ is that it can be used on disks in a pool that are not
the same size. Why then did I explicitly have to use the -f switch to force a
creation of a RAIDZ vdev on a salad of 2 and 4GB disks?
Not sure exactly which blog entry but you might be confused that
stripes can be of different sizes (not different sized disks).  The
man page for zpool specifically warns that "An attempt to combine
redundant and non-redundant storage in a single pool,  or  to
mix  disks  and  files, results in an error unless -f is specified.".



-- 
Just me,
Wire ...

> AFAIK, no.  The "attach" semantics only works for
> adding mirrors.
> Would be nice if that can be overloaded for RAIDZ.
Sure would be. 
> Not sure exactly which blog entry but you might be
> confused that
> stripes can be of different sizes (not different
> sized disks).  The
> man page for zpool specifically warns that "An
> attempt to combine
> redundant and non-redundant storage in a single pool,
>  or  to
> ix  disks  and  files, results in an error unless -f
> is specified.".
Yes, the man page says that. However, it is possible to mix disks of different
sizes in a RAIDZ, and this works. Why does it work? Because RAIDZ stripes are
dynamic in size. From that I infer that disks can be any size because the
stripes can be any size. Should one or more of the smaller disks become full,
the stripe information is written across the remaining space on the rest of the
bigger disks. Or should be anyway. If you think about it, it makes sense.

Be that as it may, I remember I read that somewhere on a blog, and I do believe
it had been Mr. Bonwick''s blog, although I could be wrong.
 
 
This message posted from opensolaris.org

UNIX admin wrote:
> [Solaris 10 6/06 i86pc]
> 
> Shortly thereafter, I ran out of space on my "space" pool, but
`zfs
> list` kept reporting I still had about a GigaByte worth of free 
> space, while `zpool status` seemed to correctly report I ran out of 
> space.
Please send us the output of ''zpool status'', ''zfs
list'', and
''zfs get -r all <poolname>''.

--matt

On 9/6/06, UNIX admin <tripivceta at hotmail.com>
wrote:
> Yes, the man page says that. However, it is possible to mix disks of
different sizes in a RAIDZ, and this works. Why does it work? Because RAIDZ
stripes are dynamic in size. From that I infer that disks can be any size
because the stripes can be any size. Should one or more of the smaller disks
become full, the stripe information is written across the remaining space on the
rest of the bigger disks. Or should be anyway. If you think about it, it makes
sense.
You are right that it can be done in theory.  The practical difficulty
is to do it efficiently.  My raidz pool tend to have a strong
preference to utilize all the disks so disks with different sizes
might result in smaller column widths later.  I''m not sure if that
spells any performance issues but at least the capacity difference
between the 2 largest disk will not be usable.

Is maximising use of capacity of all available disk in the pipeline
for RAID-Z or is it to be avoided intentionally?


-- 
Just me,
Wire ...