zfs discuss - Jan 2007 - UFS on zvol: volblocksize and maxcontig

Hi all!

First off, if this has been discussed, please point me in that 
direction. I have searched high and low and really can''t find much info
on the subject.

We have a large-ish (200gb) UFS file system on a Sun Enterprise 250 that 
is being shared with samba (lots of files, mostly random IO). OS is 
Solaris 10u3. Disk set  is 7x36gb 10k scsi, 4 internal 3 external.

For several reasons we currently need to stay on UFS and can''t switch
to
ZFS proper. So instead we have opted to do UFS on a zvol using raid-z, 
in lieu of UFS on SVM using raid5 (we want/need raid protection). This 
decision was made because of the ease of disk set portability of zpools, 
and also the [assumed] performance benefit vs SVM.

Anyways, I''ve been pondering the volblocksize parameter, and trying to 
figure out how it interacts with UFS. When the zvol was setup, I took 
the default 8k size. Since UFS uses an 8k blocksize, this seemed to be a 
reasonable choice. I''ve been thinking more about it lately, and have 
also read that UFS will do R/W in bigger than 8k blocks when it can, up 
to maxcontig (default of 16, ie 128k).

This presented me with several questions: Would a volblocksize of 128k 
and maxcontig 16 provide better UFS performance? Overall, or only in 
certain situations (ie only for sequential IO)? Would increasing the 
maxcontig beyond 16 make any difference (good, bad or indifferent) if 
the underlying device is limited to 128k blocks?

What exactly does volblocksize control? My observations thus far 
indicate that it simply sets a max block size for the [virtual] zvol 
device. Changing volblocksize does NOT seem to have an impact on IOs to 
the underlying physical disks, which always seem to float in the 50-110k 
range). How does volblocksize affect IO that is not of a set block size?

Finally, why does volblocksize only affects raidz and mirror devices? It 
seems to have no effect on ''simple'' devices, even though I
presume
striping is still used there. That is also assuming that volblocksize 
interacts with striping.

Any answers or input is greatly appreciated.

Thanks much!
-Brian

-- 
---------------------------------------------------
Brian H. Nelson         Youngstown State University
System Administrator   Media and Academic Computing
              bnelson[at]cis.ysu.edu
---------------------------------------------------

Brian H. Nelson wrote:
> For several reasons we currently need to stay on UFS and can''t
switch to
> ZFS proper. So instead we have opted to do UFS on a zvol using raid-z, 
Can you state what those reasons are please ?

I know that isn''t answering the question you are asking but it is worth
making sure you have the correct info.

I''d also like to understand why UFS works for you but ZFS as a 
filesystem does not.

-- 
Darren J Moffat

Darren J Moffat wrote:
> Brian H. Nelson wrote:
>> For several reasons we currently need to stay on UFS and can''t
switch
>> to ZFS proper. So instead we have opted to do UFS on a zvol using 
>> raid-z, 
>
> Can you state what those reasons are please ?
>
> I know that isn''t answering the question you are asking but it is 
> worth making sure you have the correct info.
>
> I''d also like to understand why UFS works for you but ZFS as a 
> filesystem does not.
>
I knew someone would ask that :)

The primary reason is that our backup software (EMC/Legato Networker 
7.2) does not appear to support zfs. We don''t have the funds currently 
to upgrade to the new version that does.

The other reason is that the machine has been around for years, already 
using UFS and quotas extensively. Over winter break we had time to 
upgrade to Solaris 10 and migrate the volume from svm to zvol, but not 
much more.There are a few thousand users on the machine. The thought of 
transitioning to that many zfs ''partitions'' in order to have
per-user
quotas seemed daunting, not to mention the administrative re-training 
needed (edquota doesn''t work. du is reporting 3000 filesystems?! etc).

IMO, the quota-per-file-system approach seems inconvenient when you get 
past a handful of file systems. Unless I''m really missing something, it
just seems like a nightmare to have to deal with such a ridiculous 
number of file systems.

-Brian

-- 
---------------------------------------------------
Brian H. Nelson         Youngstown State University
System Administrator   Media and Academic Computing
              bnelson[at]cis.ysu.edu
---------------------------------------------------

>The other reason is that the machine has been around for years, already 
>using UFS and quotas extensively. Over winter break we had time to 
>upgrade to Solaris 10 and migrate the volume from svm to zvol, but not 
>much more.There are a few thousand users on the machine. The thought of 
>transitioning to that many zfs ''partitions'' in order to
have per-user
>quotas seemed daunting, not to mention the administrative re-training 
>needed (edquota doesn''t work. du is reporting 3000 filesystems?!
etc).
I''m assuming "df"?

I think that the problem you are describing is a symptom of how
existing tools and methods fall apart when confronted with
huge numbers of filesystems, but only because more information
if presented by "df" than you did before.

I''d love to have an option to df which only reported pools, not
filesystems.  (rather than having to type "df -F ufs; zpool list")

The same problem exists with automounted home directories
(but only active directories are shown, again this is something
ZFS may want to emulate)
>IMO, the quota-per-file-system approach seems inconvenient when you get 
>past a handful of file systems. Unless I''m really missing
something, it
>just seems like a nightmare to have to deal with such a ridiculous 
>number of file systems.
Why?  What additional per-filesystem overhead from a maintenance perspective
are you seeing?

Casper

Brian H. Nelson wrote:
> IMO, the quota-per-file-system approach seems inconvenient when you get
> past a handful of file systems. Unless I''m really missing
something, it
> just seems like a nightmare to have to deal with such a ridiculous
> number of file systems.
Seconded -- is there any chance BSD-style quotas will be implemented in
ZFS?

I notice there''s an RFE:

	http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6501037

Jim

Casper.Dik at Sun.COM wrote:
> *snip*
>> IMO, the quota-per-file-system approach seems inconvenient when you get
>> past a handful of file systems. Unless I''m really missing
something, it
>> just seems like a nightmare to have to deal with such a ridiculous 
>> number of file systems.
>>     
>
> Why?  What additional per-filesystem overhead from a maintenance
perspective
> are you seeing?
>
> Casper
>   
The obvious example would be /var/mail . UFS quotas are easy. Doing the 
same thing with ZFS would be (I think) impossible. You would have to 
completely convert and existing system to a maildir or home directory 
mail storage setup.

Other file-system-specific software could also have issues. Networker 
for instance does backups per filesystem. In that situation I could then 
possibly have ~3000 backup sets DAILY for a single machine (worst case, 
that each file system has changes). Granted, that may not be better or 
worse, just ''different'' and not what I''m used to. On
the other hand, I
could certainly see where that could add a ton of overhead to backup 
processing.

Don''t get me wrong, zfs quotas are a good thing, and could certainly be
useful in many situations. I just don''t think I agree that they are a 
one to one replacement for ufs quotas in terms of usability in all 
situations.

-Brian

-- 
---------------------------------------------------
Brian H. Nelson         Youngstown State University
System Administrator   Media and Academic Computing
              bnelson[at]cis.ysu.edu
---------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070126/98c25961/attachment.html>

zfs-discuss-bounces at opensolaris.org wrote on 01/26/2007 12:20:17 PM:
> Casper.Dik at Sun.COM wrote:
> *snip*
> IMO, the quota-per-file-system approach seems inconvenient when you get
> past a handful of file systems. Unless I''m really missing
something, it
> just seems like a nightmare to have to deal with such a ridiculous
> number of file systems.
>
>
> Why?  What additional per-filesystem overhead from a maintenance
perspective
> are you seeing?
>
> Casper
>
> The obvious example would be /var/mail . UFS quotas are easy. Doing
> the same thing with ZFS would be (I think) impossible. You would
> have to completely convert and existing system to a maildir or home
> directory mail storage setup.
>
> Other file-system-specific software could also have issues.
> Networker for instance does backups per filesystem. In that
> situation I could then possibly have ~3000 backup sets DAILY for a
> single machine (worst case, that each file system has changes).
> Granted, that may not be better or worse, just
''different'' and not
> what I''m used to. On the other hand, I could certainly see where
> that could add a ton of overhead to backup processing.
>
> Don''t get me wrong, zfs quotas are a good thing, and could
certainly
> be useful in many situations. I just don''t think I agree that they
> are a one to one replacement for ufs quotas in terms of usability in
> all situations.
Yes, there is an RFE out there for this that has been "dispatched"?. 
In
many cases the zfs quotas work very well and are actually a godsend (after
getting over the initial shock of seeing screens of df output) but they
fail to cover usage when a filesystem or directory tree must be shared by
multiple users where each users needs to have limits to what disk space
they may use -- think department folders, or your example of mail.  The RFE
does not go into details about HOW this would be done when implemented.
User level quotas don''t need to exactly match ufs quotas -- they can be
rethunk for zfs.  Are zfs style user quotas:

per zfs fs?
per zfs fs and all of their children (recursive)?
affected by snapshots data usage?
applied to lists of fs and summed (
username:100,000:/tank/home/username;/tank/departments/usersdepartment  #
allow 100,000 bytes to be used in total between these two unrelated
filesystems by this user)...

I have faith that user quotas are going to come sometime, these how
questions are interesting to me...


-Wade Stuart

On 1/26/07, Darren J Moffat <Darren.Moffat at sun.com>
wrote:
> Brian H. Nelson wrote:
> > For several reasons we currently need to stay on UFS and
can''t switch to
> > ZFS proper. So instead we have opted to do UFS on a zvol using raid-z,
>
> Can you state what those reasons are please ?
>
> I know that isn''t answering the question you are asking but it is
worth
> making sure you have the correct info.
>
> I''d also like to understand why UFS works for you but ZFS as a
> filesystem does not.
Samba does not currently support ZFS ACLs.  This thread caught my eye
as I just recently considered a similar solution.  Support is being
worked on though, apparently, so I can wait:

http://lists.samba.org/archive/samba-technical/2007-January/051123.html

-- 
Eric Enright

Eric Enright wrote:
> Samba does not currently support ZFS ACLs.
Yes, but this just means you can''t get/set your ACLs from a CIFS
client.  ACLs will be enforced just fine once set locally on the
server; you may also be able to get/set them from an NFS client.
You may know this, but I know some are confused by this and think
you lose ACL protection.

Rob T

On 1/26/07, Robert Thurlow <robert.thurlow at sun.com>
wrote:
> Eric Enright wrote:
>
> > Samba does not currently support ZFS ACLs.
>
> Yes, but this just means you can''t get/set your ACLs from a CIFS
> client.  ACLs will be enforced just fine once set locally on the
> server; you may also be able to get/set them from an NFS client.
> You may know this, but I know some are confused by this and think
> you lose ACL protection.
Quite right.

Getting them set poses a problem for my specific case, unfortunately.

-- 
Eric Enright

I hope there will be consideration given to providing compatibility with UFS
quotas
(except that inode limits would be ignored).  At least to the point of having

edquota(1m)
quot(1m)
quota(1m)
quotactl(7i)
repquota(1m)
rquotad(1m)

and possibly quotactl(7i) work with zfs (with the exception previously
mentioned).
OTOH, quotaon(1m)/quotaoff(1m)/quotacheck(1m) may not be needed for support of
per-user quotas in zfs (since it will presumably have its own ways of enabling
these, and
will simply never mess up?)

None of which need preclude new interfaces with greater functionality (like both
user and group quotas), but where there is similar functionality, IMO it would
be
easier for a lot of folks if quota maintenance (esp. edquota and reporting)
could
be done the same way for ufs and zfs.
 
 
This message posted from opensolaris.org