zfs discuss - Aug 2006 - ZFS performance using slices vs. entire disk?

I know this is going to sound a little vague but...

A coworker said he read somewhere that ZFS is more efficient if you 
configure pools from entire disks instead of just slices of disks. I''m 
curious if there is any merit to this?

The use case that we had been discussing was something to the effect of 
building a 2 disk system, install the OS on slice 0 of disk 0 and make 
the rest of the disk available for 1/2 of a zfs mirror. Then disk 1 
would probably be partitioned the same, but the only thing active would 
be the other 1/2 of a zfs mirror.

Now clearly there is a contention issue between the OS and the data 
partition, which would be there if SVM mirrors were used instead. But 
besides this, is zfs any less efficient with just using a portion of a 
disk versus the entire disk?

On Wed, 2 Aug 2006, Joseph Mocker wrote:
> The use case that we had been discussing was something to the effect of
> building a 2 disk system, install the OS on slice 0 of disk 0 and make the
> rest of the disk available for 1/2 of a zfs mirror. Then disk 1 would
probably
> be partitioned the same, but the only thing active would be the other 1/2
of a
> zfs mirror.
Why wouldn''t you mirror (using SVM) the OS slice on disk 1 too?

Sorry, can''t answer the ZFS bit of the question...

-- 
Rich Teer, SCNA, SCSA, OpenSolaris CAB member

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-group.com/rich

Joseph Mocker wrote:
> I know this is going to sound a little vague but...
>
> A coworker said he read somewhere that ZFS is more efficient if you 
> configure pools from entire disks instead of just slices of disks.
I''m
> curious if there is any merit to this?

If the entire disk is used in a zpool then the disk cache can, and in 
most cases is, enabled. This speeds operations up quite a bit in some 
scenarios.

Hello Joseph,

Thursday, August 3, 2006, 2:02:28 AM, you wrote:

JM> I know this is going to sound a little vague but...

JM> A coworker said he read somewhere that ZFS is more efficient if you 
JM> configure pools from entire disks instead of just slices of disks.
I''m
JM> curious if there is any merit to this?

JM> The use case that we had been discussing was something to the effect of
JM> building a 2 disk system, install the OS on slice 0 of disk 0 and make
JM> the rest of the disk available for 1/2 of a zfs mirror. Then disk 1 
JM> would probably be partitioned the same, but the only thing active would
JM> be the other 1/2 of a zfs mirror.

JM> Now clearly there is a contention issue between the OS and the data 
JM> partition, which would be there if SVM mirrors were used instead. But 
JM> besides this, is zfs any less efficient with just using a portion of a
JM> disk versus the entire disk?

ZFS will try to enable write cache if whole disks is given.

Additionally keep in mind that outer region of a disk is much faster.
So if you want to put OS and then designate rest of the disk for
application then probably putting ZFS on a slice beginning on cyl 0 is
best in most scenarios.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

> ZFS will try to enable write cache if whole disks is given.
> 
> Additionally keep in mind that outer region of a disk is much faster.
And it''s portable.  If you use whole disks, you can export the
pool from one machine and import it on another.  There''s no way
to export just one slice and leave the others behind...

Jeff

> is zfs any less efficient with just using a portion of a 
> disk versus the entire disk?
As others mentioned, if we''re given a whole disk (i.e. no slice
is specified) then we can safely enable the write cache.

One other effect -- probably not huge -- is that the block placement
algorithm is most optimal for an outer-to-inner track diameter ratio
of about 2:1, which reflects typical platters.  To quote the source:

http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/fs/zfs/metaslab.c#m
etaslab_weight

        /*
         * Modern disks have uniform bit density and constant angular velocity.
         * Therefore, the outer recording zones are faster (higher bandwidth)
         * than the inner zones by the ratio of outer to inner track diameter,
         * which is typically around 2:1.  We account for this by assigning
         * higher weight to lower metaslabs (multiplier ranging from 2x to 1x).
         * In effect, this means that we''ll select the metaslab with
the most
         * free bandwidth rather than simply the one with the most free space.
         */

But like I said, the effect isn''t huge -- the high-order bit that we
have a preference for low LBAs.  It''s a second-order optimization
to bias the allocation based on the maximum free bandwidth, which is
currently based on an assumption about physical disk construction.
In the future we''ll do the smart thing and compute each
metaslab''s
allocation bias based on its actual observed bandwidth.

Jeff

Jeff Bonwick wrote:
>>is zfs any less efficient with just using a portion of a 
>>disk versus the entire disk?
>>    
>>
>
>As others mentioned, if we''re given a whole disk (i.e. no slice
>is specified) then we can safely enable the write cache.
>  
>
With all of the talk about performance problems due to
ZFS doing a sync to force the drives to commit to data
being on disk, how much of a benefit is this - especially
for NFS?

Also, if I was lucky enough to have a working prestoserv
card around, would ZFS be able to take advantage of
that at all?

Darren

> With all of the talk about performance problems due to
> ZFS doing a sync to force the drives to commit to data
> being on disk, how much of a benefit is this - especially
> for NFS?
It depends.  For some drives it''s literally 10x.
> Also, if I was lucky enough to have a working prestoserv
> card around, would ZFS be able to take advantage of
> that at all?
I''m working on the general lack-of-NVRAM-in-servers problem.
As for using presto, I don''t think it''d be too hard. 
We''ve
already structured the code so that allocating intent log
blocks from a different set of vdevs would be straightforward.
It''s probably a week''s work to define the new metaslab class,
new vdev type, and modify the ZIL to use it.

Jeff

On Aug 3, 2006, at 8:17 AM, Jeff Bonwick wrote:
>> ZFS will try to enable write cache if whole disks is given.
>>
>> Additionally keep in mind that outer region of a disk is much faster.
>
> And it''s portable.  If you use whole disks, you can export the
> pool from one machine and import it on another.  There''s no way
> to export just one slice and leave the others behind...
I got the impression that the export command exported the contents
of the pool, not the underlying physical structure (minus the rust ;o).
Seems that I was wrong.

This also implies that the disk that receives the exported pool
has to be the same size as the exported disk, or can it be bigger ???



Kaiser Jasse -- Authorized Stealth Oracle

The axioms of wisdom:
1. You can''t outstubborn a cat
2. You can''t conquer the universe without the knowledge of FORTRAN
3. In the Unix realm, 10% of work fixes 90% of the problems

Robert Milkowski <rmilkowski at task.gda.pl> writes:
> Additionally keep in mind that outer region of a disk is much faster.
> So if you want to put OS and then designate rest of the disk for
> application then probably putting ZFS on a slice beginning on cyl 0 is
> best in most scenarios.
This has the additional advantage that with the advent of ZFS boot, you can
simply move / to a zfs file system and extend slice 0 to cover the whole
disk.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

> > And it''s portable.  If you use whole disks, you can export
the
> > pool from one machine and import it on another.  There''s no
way
> > to export just one slice and leave the others behind...
> 
> I got the impression that the export command exported the contents
> of the pool, not the underlying physical structure (minus the rust ;o).
> Seems that I was wrong.
Correct.  You could export a pool that existed on just one of the
slices, but it would be difficult to make that useful.  You couldn''t
for
example put slice 1 in a new machine and leave slice 4 here.
> This also implies that the disk that receives the exported pool
> has to be the same size as the exported disk, or can it be bigger ???
Hmm?  An ''exported'' pool remains on the same media, but is no
longer
being used by the host.  So the media can be transported and connected
(imported) elsewhere.  

It doesn''t move the data to new meda (that would be zfs send or the
like).

-- 
Darren Dunham                                           ddunham at taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

On Aug 3, 2006, at 5:14 PM, Darren Dunham wrote:
>>> And it''s portable.  If you use whole disks, you can export
the
>>> pool from one machine and import it on another.  There''s
no way
>>> to export just one slice and leave the others behind...
>>
>> I got the impression that the export command exported the contents
>> of the pool, not the underlying physical structure (minus the  
>> rust ;o).
>> Seems that I was wrong.
>
> Correct.  You could export a pool that existed on just one of the
> slices, but it would be difficult to make that useful.  You  
> couldn''t for
> example put slice 1 in a new machine and leave slice 4 here.
>
>> This also implies that the disk that receives the exported pool
>> has to be the same size as the exported disk, or can it be bigger ???
>
> Hmm?  An ''exported'' pool remains on the same media, but
is no longer
> being used by the host.  So the media can be transported and connected
> (imported) elsewhere.
>
> It doesn''t move the data to new meda (that would be zfs send or
the
> like).
Oops, seems that I swapped the commands ''export'' and
''send'' in my head.




Kaiser Jasse -- Authorized Stealth Oracle

The axioms of wisdom:
1. You can''t outstubborn a cat
2. You can''t conquer the universe without the knowledge of FORTRAN
3. In the Unix realm, 10% of work fixes 90% of the problems

Ahh, interesting information. Thanks folks, I''m have a better 
understanding of this now.

  --joe

Jeff Bonwick wrote:
>> is zfs any less efficient with just using a portion of a 
>> disk versus the entire disk?
>>     
>
> As others mentioned, if we''re given a whole disk (i.e. no slice
> is specified) then we can safely enable the write cache.
>
> One other effect -- probably not huge -- is that the block placement
> algorithm is most optimal for an outer-to-inner track diameter ratio
> of about 2:1, which reflects typical platters.  To quote the source:
>
>
http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/fs/zfs/metaslab.c#m
> etaslab_weight
>
>         /*
>          * Modern disks have uniform bit density and constant angular
velocity.
>          * Therefore, the outer recording zones are faster (higher
bandwidth)
>          * than the inner zones by the ratio of outer to inner track
diameter,
>          * which is typically around 2:1.  We account for this by assigning
>          * higher weight to lower metaslabs (multiplier ranging from 2x to
1x).
>          * In effect, this means that we''ll select the metaslab
with the most
>          * free bandwidth rather than simply the one with the most free
space.
>          */
>
> But like I said, the effect isn''t huge -- the high-order bit that
we
> have a preference for low LBAs.  It''s a second-order optimization
> to bias the allocation based on the maximum free bandwidth, which is
> currently based on an assumption about physical disk construction.
> In the future we''ll do the smart thing and compute each
metaslab''s
> allocation bias based on its actual observed bandwidth.
>
> Jeff
>
>

Folks,

I realize this thread has run its course, but I''ve got a variant of
the original question:  What performance problems or anomalies might
one see if mixing both whole disks _and_ slices within the same pool?

I have in mind some Sun boxes (V440, T2000, X4200) with four internal
drives.  Typically, one would setup root/swap/var as an SVM/UFS mirror
on one pair of drives, and the second pair as a mirrored ZFS pair.

However, on the typical 72GB drives, much of that first pair (maybe half
or more) would not be needed for system+swap.  Suppose we use 24GB of
the first pair as our SVM/UFS system mirror, with the free space left
in slice 5 of each drive.  We could then make maximum use of the remaining
space as a zfs pool via something like:

  zpool create mirror c0t2d0 c0t3d0 mirror c0t0d0s5 c0t1d0s5

Is this allowed?  Is it stupid?  Will performance be so bad/bizarre that
it should be avoided at all costs?  Anybody tried it?

BTW, in my experience the OS/swap/var filesystems get very little I/O
unless RAM is in short supply.  So on a well-sized system, there should
be little contention between the "system" portion and "data"
portion
of those shared disks (regardless of whether or not ZFS is in use).

Thanks and regards,

Marion

On Thu, Aug 03, 2006 at 10:24:12AM -0700, Marion Hakanson
wrote:
> 
>   zpool create mirror c0t2d0 c0t3d0 mirror c0t0d0s5 c0t1d0s5
> 
> Is this allowed?  Is it stupid?  Will performance be so bad/bizarre that
> it should be avoided at all costs?  Anybody tried it?
> 
Yes, it''s allowed, but it''s definitely not an ideal solution. 
Unless
you manually tweak the write cache (as mentioned elsewhere, this could
be bad for other slices), you''ll end up with asymmetric performance on
these two vdevs.  The current ZFS allocation and I/O policies don''t
take
latency into account (they''re based solely on capacity), and so a
slower
vdev will slow down the entire pool.

Future work will make ZFS behave better in the face of asymmetric
performance, but in general this will still be a non-ideal
configuration.  You''ll be fighting with other filesystems for control
of
the underlying disk, you won''t be able to use hot spares effectively,
and future FMA work (such as reading SMART data and predictive analysis)
may not work well (if at all).

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Eric Schrock wrote:
> On Thu, Aug 03, 2006 at 10:24:12AM -0700, Marion Hakanson wrote:
>   
>>   zpool create mirror c0t2d0 c0t3d0 mirror c0t0d0s5 c0t1d0s5
>>
>> Is this allowed?  Is it stupid?  Will performance be so bad/bizarre
that
>> it should be avoided at all costs?  Anybody tried it?
>>
>>     
>
> Yes, it''s allowed, but it''s definitely not an ideal
solution.  Unless
> you manually tweak the write cache (as mentioned elsewhere, this could
> be bad for other slices), you''ll end up with asymmetric
performance on
> these two vdevs.  The current ZFS allocation and I/O policies
don''t take
> latency into account (they''re based solely on capacity), and so a
slower
> vdev will slow down the entire pool.
>   
Is the fact that there are more spindles in general going to help at
all? So in this case he''s got a pool consisting of two mirrored
stripes.
Isn''t performance still going to be better than if he dropped off the
slower slices/stripe and had a single mirror? Won''t the system be able
to handle more i/os in general by taking advantage of the additional
bandwidth provided by each controller?

For example, while one block is being written to the slower stripe, it
may also be able to write another block to the faster stripe concurrently?

  --joe

Darren:

 > > With all of the talk about performance problems due to
 > > ZFS doing a sync to force the drives to commit to data
 > > being on disk, how much of a benefit is this - especially
 > > for NFS?

I would not call those things as problems, more like setting 
proper expectations.

My understanding is that enabling write cache helps by
providing I/O concurrency for drives that do not implement
other form of Command Queuing. In other cases, WCE should
not buy much if anything. I''d be interested in analysing any 
cases that shows otherwise...

-r