zfs discuss - Jul 2006 - Re: Poor performance on NFS-exported ZFS volumes

Patrick Bachmann:
> Hey Bill,
>
> Bill Sommerfeld wrote:
>> Overly wide raidz groups seems to be an unfenced hole that people new
to
>> ZFS fall into on a regular basis.
>>
>> The man page warns against this but that doesn''t seem to be
sufficient.
>>
>> Given that zfs has relatively few such traps, perhaps large raidz
groups
>> ought to be implicitly split up absent a "Yes, I want to be
stupid"
>> flag..
>
> IMHO it is sufficient to just document this best-practice.
I disagree.  The documentation has to AT LEAST state that more than 9
disks gives poor performance.  I did read that raidz should use 3-9 disks
in the docs but it doesn''t say WHY, so of course I went ahead and used
12 disks.

When I say I disagree, I mean this has to be documented in the standard
docs (man pages) not some best-practices guide on some wiki.

But really I disagree that this needs documentation.  So much of zfs is
meant to be automatic, now we''re back to worrying about stripe width?
(Or maybe that''s not the problem but it sure is the same type of manual
administration.)  I may have 12 disks and it simply does not make sense
(for my theoretical configuration) to split them up into two pools.  I
would then have to worry about sizing each pool correctly.  zfs is supposed
to fix that problem.

-frank

Frank Cusack wrote:
> Patrick Bachmann:
>> Hey Bill,
>>
>> Bill Sommerfeld wrote:
>>> Overly wide raidz groups seems to be an unfenced hole that people
new to
>>> ZFS fall into on a regular basis.
>>>
>>> The man page warns against this but that doesn''t seem to
be sufficient.
>>>
>>> Given that zfs has relatively few such traps, perhaps large raidz
groups
>>> ought to be implicitly split up absent a "Yes, I want to be
stupid"
>>> flag..
>>
>> IMHO it is sufficient to just document this best-practice.
> 
> I disagree.  The documentation has to AT LEAST state that more than 9
> disks gives poor performance.  I did read that raidz should use 3-9 disks
> in the docs but it doesn''t say WHY, so of course I went ahead and
used
> 12 disks.
> 
> When I say I disagree, I mean this has to be documented in the standard
> docs (man pages) not some best-practices guide on some wiki.
> 
> But really I disagree that this needs documentation.  So much of zfs is
> meant to be automatic, now we''re back to worrying about stripe
width?
> (Or maybe that''s not the problem but it sure is the same type of
manual
> administration.)  I may have 12 disks and it simply does not make sense
> (for my theoretical configuration) to split them up into two pools.  I
> would then have to worry about sizing each pool correctly.  zfs is supposed
> to fix that problem.
> 
This may just be a matter of wording, but you wouldn''t have to split it
up into two pools.  You could use two smaller raidz vdevs within the 
same pool.

-- Rich.

Hey Frank,

Frank Cusack wrote:
> Patrick Bachmann:
>> IMHO it is sufficient to just document this best-practice.
> 
> I disagree.  The documentation has to AT LEAST state that more than 9
> disks gives poor performance.  I did read that raidz should use 3-9 disks
> in the docs but it doesn''t say WHY, so of course I went ahead and
used
> 12 disks.
The man pages says "The recommended number [of devices] is between 3 
and 9". At the time of the discussion the "ZFS Admin Guide"
didn''t
mention this at all and the result of this discussion is, that now 
it says: "If you are creating a RAID-Z configuration with many 
disks, as an example, a RAID-Z configuration with 14 disks is better 
split up into a two 7-disk groupings. RAID-Z configurations with 
single-digit groupings of disks should perform better."
> When I say I disagree, I mean this has to be documented in the standard
> docs (man pages) not some best-practices guide on some wiki.
To me "documenting a best-practice" does not imply that it is just 
spelled out in some random wiki or a BluePrint but rather that it is 
written down "somewhere" and that this "best-practice" is
not just
documented in the mailing-lists archives, which in some other 
communities seem to be the only documentation available.
> But really I disagree that this needs documentation.  So much of zfs is
> meant to be automatic, now we''re back to worrying about stripe
width?
> (Or maybe that''s not the problem but it sure is the same type of
manual
> administration.)  I may have 12 disks and it simply does not make sense
> (for my theoretical configuration) to split them up into two pools.  I
> would then have to worry about sizing each pool correctly.  zfs is supposed
> to fix that problem.
Richard already pointed out that you should split the devices into a 
number of vdevs and not pools.
How is ZFS going to know what gives the best performance on your 
systems config? There are a lot of things you know better off-hand 
about your system, otherwise you need to do some benchmarking, which 
ZFS would have to do too, if it was to give you the best performing 
config.
Oh, and performance isn''t the only objective. See the threads 
started by Richard Elling in the past couple of weeks on this 
mailinglist.

If no one has done it yet, I''ll file a bug against the zpool(1M) man 
page to get the performance concern included. But to me the "ZFS 
Admin Guide" seemed to be a required reading. You might want to 
check it out.

Greetings,

Patrick

On Fri, Jul 28, 2006 at 02:14:50PM +0200, Patrick Bachmann
wrote:
> systems config? There are a lot of things you know better off-hand 
> about your system, otherwise you need to do some benchmarking, which 
> ZFS would have to do too, if it was to give you the best performing 
> config.
How hard would it be to write a tool like that?  Something along the
lines of:

zpool bench raidz disk1 disk2 ... diskN

Let ZFS figure out the best way to set up your disks for you and tell
you how it should be laid out (and even offer a "just do it" flag that
will let it automatically create the pool depending on how it sees it
best)?

Yes, I know my hardware, and so perhaps I wouldn''t need such a tool,
but
there are a lot of people out there who don''t really know the best way
to use their hardware for the best performance.  This would be an
outstanding tool for them.

-brian

Brian Hechinger wrote:
> On Fri, Jul 28, 2006 at 02:14:50PM +0200, Patrick Bachmann wrote:
>> systems config? There are a lot of things you know better off-hand 
>> about your system, otherwise you need to do some benchmarking, which 
>> ZFS would have to do too, if it was to give you the best performing 
>> config.
> 
> How hard would it be to write a tool like that?  Something along the
> lines of:
> 
> zpool bench raidz disk1 disk2 ... diskN
> 
> Let ZFS figure out the best way to set up your disks for you and tell
> you how it should be laid out (and even offer a "just do it" flag
that
> will let it automatically create the pool depending on how it sees it
> best)?
The problem is that there are at least 3 knobs to turn (space, RAS, and
performance) and they all interact with each other.
> Yes, I know my hardware, and so perhaps I wouldn''t need such a
tool, but
> there are a lot of people out there who don''t really know the best
way
> to use their hardware for the best performance.  This would be an
> outstanding tool for them.
I''ve got an idea I''m prototyping, but it isn''t
ZFS-specific, so I don''t
expect to tie it directly to ZFS.  Stay tuned...
  -- richard

On July 28, 2006 2:14:50 PM +0200 Patrick Bachmann <patrickbachmann at
gmx.net> wrote:
> Richard already pointed out that you should split the devices into a number
of vdevs and not
> pools.
I missed that.  I guess I also didn''t know what a vdev is, guess I know
even less about this "zfs thing" than I thought. :-)

thanks
-frank

On July 28, 2006 9:09:58 AM -0400 Brian Hechinger <wonko at 4amlunch.net>
wrote:
> On Fri, Jul 28, 2006 at 02:14:50PM +0200, Patrick Bachmann wrote:
>> systems config? There are a lot of things you know better off-hand
>> about your system, otherwise you need to do some benchmarking, which
>> ZFS would have to do too, if it was to give you the best performing
>> config.
>
> How hard would it be to write a tool like that?  Something along the
> lines of:
>
> zpool bench raidz disk1 disk2 ... diskN
bonnie++?

-frank

Richard Elling wrote:
>>
>> How hard would it be to write a tool like that?  Something along the
>> lines of:
>>
>> zpool bench raidz disk1 disk2 ... diskN
>>
>> Let ZFS figure out the best way to set up your disks for you and tell
>> you how it should be laid out (and even offer a "just do it"
flag that
>> will let it automatically create the pool depending on how it sees it
>> best)?
>
> The problem is that there are at least 3 knobs to turn (space, RAS, and
> performance) and they all interact with each other.
Good point. then how about something more like

    zpool bench raidz favor space disk1 ... diskN
    zpool bench raidz favor performance disk1 .. diskN

That is, tell the analyzer which knob you are most interested in.

Joseph Mocker wrote:
> Richard Elling wrote:
>> The problem is that there are at least 3 knobs to turn (space, RAS, and
>> performance) and they all interact with each other.
> 
> Good point. then how about something more like
> 
>    zpool bench raidz favor space disk1 ... diskN
>    zpool bench raidz favor performance disk1 .. diskN
> 
> That is, tell the analyzer which knob you are most interested in.
I wish it was that easy.  If I optimize for space, I''ll always get a
big,
fat RAID-0.  If I optimize for RAS, I''ll get multi paned (N-way)
mirror.
The tool has to be able to handle the spectrum in between those extremes.
  -- richard

On Fri, Jul 28, 2006 at 02:02:13PM -0700, Richard Elling
wrote:
> Joseph Mocker wrote:
> >Richard Elling wrote:
> >>The problem is that there are at least 3 knobs to turn (space, RAS,
and
> >>performance) and they all interact with each other.
> >
> >Good point. then how about something more like
> >
> >   zpool bench raidz favor space disk1 ... diskN
> >   zpool bench raidz favor performance disk1 .. diskN
> >
> >That is, tell the analyzer which knob you are most interested in.
> 
> I wish it was that easy.  If I optimize for space, I''ll always get
a big,
> fat RAID-0.  If I optimize for RAS, I''ll get multi paned (N-way)
mirror.
> The tool has to be able to handle the spectrum in between those extremes.
Look closer at the format of that command:

zpool bench *RAIDZ* blah....

RAID-0 isn''t an option, find the best parameters for what is specified,
in this case we are constrained to RAIDZ.

-brian

Brian Hechinger wrote:
> On Fri, Jul 28, 2006 at 02:02:13PM -0700, Richard Elling wrote:
>> Joseph Mocker wrote:
>>> Richard Elling wrote:
>>>> The problem is that there are at least 3 knobs to turn (space,
RAS, and
>>>> performance) and they all interact with each other.
>>> Good point. then how about something more like
>>>
>>>   zpool bench raidz favor space disk1 ... diskN
>>>   zpool bench raidz favor performance disk1 .. diskN
>>>
>>> That is, tell the analyzer which knob you are most interested in.
>> I wish it was that easy.  If I optimize for space, I''ll always
get a big,
>> fat RAID-0.  If I optimize for RAS, I''ll get multi paned
(N-way) mirror.
>> The tool has to be able to handle the spectrum in between those
extremes.
> 
> Look closer at the format of that command:
> 
> zpool bench *RAIDZ* blah....
> 
> RAID-0 isn''t an option, find the best parameters for what is
specified,
> in this case we are constrained to RAIDZ.
Given N disks, what is the best raidz configuration is also not a
trivial question to answer.  The spectrum is narrowed to something
between N-way and (int)N/3 3-way plus at least one spare.
  -- richard