zfs discuss - Sep 2007 - ext3 on zvols journal performance pathologies?

I''ve been seeing read and write performance pathologies with Linux
ext3 over iSCSI to zvols, especially with small writes. Does running
a journalled filesystem to a zvol turn the block storage into swiss
cheese? I am considering serving ext3 journals (and possibly swap
too) off a raw, hardware-mirrored device. Before I do (and I''ll
write up any results) I''d like to know if anyone tried/addressed
this already.

The lack of tools to analyse ZFS fragmentation means I''m somewhat
in the dark, so I''m just likely to suck it and see.

JG

Joshua Goodall wrote:
> I''ve been seeing read and write performance pathologies with Linux
> ext3 over iSCSI to zvols, especially with small writes. Does running
> a journalled filesystem to a zvol turn the block storage into swiss
> cheese? I am considering serving ext3 journals (and possibly swap
> too) off a raw, hardware-mirrored device. Before I do (and I''ll
> write up any results) I''d like to know if anyone tried/addressed
> this already.
What evidence do you have that it is the ZVOL at fault here rather than 
say the iSCSI initiator or target or the actually physical storage 
backing the zpool or the ext3 filesystem ?  Or even the network ?

What kind of pool config do you have ?
What does the network look like ?
What does zpool iostat show you when doing the writes ?

etc etc

-- 
Darren J Moffat

Joshua Goodall wrote:
> I''ve been seeing read and write performance pathologies with Linux
> ext3 over iSCSI to zvols, especially with small writes. Does running
> a journalled filesystem to a zvol turn the block storage into swiss
> cheese? I am considering serving ext3 journals (and possibly swap
> too) off a raw, hardware-mirrored device. Before I do (and I''ll
> write up any results) I''d like to know if anyone tried/addressed
> this already.
> 
> The lack of tools to analyse ZFS fragmentation means I''m somewhat
> in the dark, so I''m just likely to suck it and see.
There are tools to analyze fragmentation, but it doesn''t seem to be a
problem.  For example,
	http://blogs.sun.com/relling/entry/zfs_i_os_in_motion

ZFS will coalesce small writes into large writes, when possible.

You should look at the latencies in the various parts of the system
to see where the bottlenecks are.  I will not claim that this is easy,
as most of the tools available do statistical sampling which may hide
the details you seek.

There have been a few iSCSI related performance bug fixes over the past
year or so.  Hopefully you''re not tripping over something that was
already fixed.
	http://bugs.opensolaris.org/view_bug.do?bug_id=6551952
	http://bugs.opensolaris.org/view_bug.do?bug_id=6496341
  -- richard

I would also suggest setting the recordsize property on the zvol when
you create it to 4k, which is, I think, the native ext3 block size.
If you don''t do this and allow ZFS to use it''s 128k default
blocksize,
then a 4k write from ext3 will turn into a 128k read/modify/write on the
ZFS side.  This could easily explain your performance issues.

Try this out and let us know what the results are.


--Bill

On Tue, Sep 11, 2007 at 11:49:40AM +1000, Joshua Goodall
wrote:
> I''ve been seeing read and write performance pathologies with Linux
> ext3 over iSCSI to zvols, especially with small writes. Does running
> a journalled filesystem to a zvol turn the block storage into swiss
> cheese? I am considering serving ext3 journals (and possibly swap
> too) off a raw, hardware-mirrored device. Before I do (and I''ll
> write up any results) I''d like to know if anyone tried/addressed
> this already.
> 
> The lack of tools to analyse ZFS fragmentation means I''m somewhat
> in the dark, so I''m just likely to suck it and see.
> 
> JG
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Tue, Sep 11, 2007 at 01:31:17PM -0700, Bill Moore
wrote:
> I would also suggest setting the recordsize property on the zvol when
> you create it to 4k, which is, I think, the native ext3 block size.
> If you don''t do this and allow ZFS to use it''s 128k
default blocksize,
> then a 4k write from ext3 will turn into a 128k read/modify/write on the
> ZFS side.  This could easily explain your performance issues.
> 
> Try this out and let us know what the results are.
A slight clarification: the default volblocksize (zvol version of
recordsize) is 8k, not 128k.  The default recordsize (not applicable to
zvols) is 128k.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock