zfs discuss - Apr 2010 - Sync Write - ZIL log performance - Feedback for ZFS developers?

Neil or somebody?  Actual ZFS developers?  Taking feedback here?   ;-)

 

While I was putting my poor little server through cruel and unusual
punishment as described in my post a moment ago, I noticed something
unexpected:

 

I expected that while I''m stressing my log device by infinite sync
writes,
my primary storage devices would also be busy(ish).  Not really busy, but
not totally idle either.  Since the primary storage is a stripe of spindle
mirrors, obviously it can handle much more sustainable throughput than the
individual log device, but the log device can respond with smaller latency.
What I noticed was this:

 

For several seconds, *only* the log device is busy.  Then it stops, and for
maybe 0.5 secs *only* the primary storage disks are busy.  Repeat, recycle.

 

I expected to see the log device busy nonstop.  And the spindle disks
blinking lightly.  As long as the spindle disks are idle, why wait for a
larger TXG to be built?  Why not flush out smaller TXG''s as long as the
disks are idle?  But worse yet . During the 1-second (or 0.5 second) that
the spindle disks are busy, why stop the log device?  (Presumably also
stopping my application that''s doing all the writing.)

 

This means, if I''m doing zillions of *tiny* sync writes, I will get the
best
performance with the dedicated log device present.  But if I''m doing
large
sync writes, I would actually get better performance without the log device
at all.  Or else . add just as many log devices as I have primary storage
devices.  Which seems kind of crazy.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100410/30460627/attachment.html>

On Sat, 10 Apr 2010, Edward Ned Harvey wrote:
> 
> For several seconds, *only* the log device is busy.? Then it stops, 
> and for maybe 0.5 secs *only* the primary storage disks are busy.? 
> Repeat, recycle.
> 
> I expected to see the log device busy nonstop.? And the spindle 
> disks blinking lightly.? As long as the spindle disks are idle, why 
> wait for a larger TXG to be built?? Why not flush out smaller TXG?s 
> as long as the disks are idle?? But worse yet ? During the 1-second 
> (or 0.5 second) that the spindle disks are busy, why stop the log 
> device?? (Presumably also stopping my application that?s doing all 
> the writing.)
What you are seeing should be expected and is good.  The intent log 
allows synchronous writes to be turned into lazy ordinary writes (like 
async writes) in the next TXG cycle.  Since the intent log is on a 
SSD, the pressure is taken off of the primary disks to serve that 
function so you will not see so many IOPS to the primary disks.
> This means, if I?m doing zillions of *tiny* sync writes, I will get 
> the best performance with the dedicated log device present.? But if 
> I?m doing large sync writes, I would actually get better performance 
> without the log device at all.? Or else ? add just as many log 
> devices as I have primary storage devices.? Which seems kind of 
> crazy.
If this is really a problem for you, then you should be able to 
somewhat resolve it by placing a smaller cap on the maximum size of a 
TXG.  Then the system will write more often.  However, the maximum 
synchronous bulk write rate will still be limited by the bandwidth of 
your intent log devices.  Huge synchronous bulk writes are pretty rare 
since usually the bottleneck is elsewhere, such as the ethernet.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Sat, Apr 10, 2010 at 11:50:05AM -0500, Bob Friesenhahn
wrote:
> Huge synchronous bulk writes are pretty rare since usually the 
> bottleneck is elsewhere, such as the ethernet.
Also, large writes can go straight to the pool, and the zil only logs
the intent to commit those blocks (ie, link them into the zfs data
structure).   I don''t recall what the threshold for this is, but I
think it''s one of those Evil Tunables.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100411/6ce8d7e7/attachment.bin>

On 04/10/10 09:28, Edward Ned Harvey wrote:
>
> Neil or somebody?  Actual ZFS developers?  Taking feedback here?   ;-)
>
>  
>
> While I was putting my poor little server through cruel and unusual 
> punishment as described in my post a moment ago, I noticed something 
> unexpected:
>
>  
>
> I expected that while I''m stressing my log device by infinite sync
> writes, my primary storage devices would also be busy(ish).  Not 
> really busy, but not totally idle either.  Since the primary storage 
> is a stripe of spindle mirrors, obviously it can handle much more 
> sustainable throughput than the individual log device, but the log 
> device can respond with smaller latency.  What I noticed was this:
>
>  
>
> For several seconds, **only** the log device is busy.  Then it stops, 
> and for maybe 0.5 secs **only** the primary storage disks are busy.  
> Repeat, recycle.
>
These are the txgs getting pushed out.
>
>  
>
> I expected to see the log device busy nonstop.  And the spindle disks 
> blinking lightly.  As long as the spindle disks are idle, why wait for 
> a larger TXG to be built?  Why not flush out smaller TXG''s as long
as
> the disks are idle?
>
Sometimes it''s more efficient to batch up requests. Less blocks are 
written. As you mentioned you weren''t stressing the system heavily.
ZFS will perform differently when under pressure. It will shorten the 
time between txgs if the data arrives quicker.
>   But worse yet ... During the 1-second (or 0.5 second) that the 
> spindle disks are busy, why stop the log device?  (Presumably also 
> stopping my application that''s doing all the writing.)
>
Yes, this has been observed by many people. There are two sides to this 
problem related to the
CPU and IO used while pushing a txg:

6806882 need a less brutal I/O scheduler
6881015 ZFS write activity prevents other threads from running in a 
timely manner

The CPU side (6881015) was fixed relatively recently in snv_129.
>  
>
> This means, if I''m doing zillions of **tiny** sync writes, I will
get
> the best performance with the dedicated log device present.  But if 
> I''m doing large sync writes, I would actually get better
performance
> without the log device at all.  Or else ... add just as many log 
> devices as I have primary storage devices.  Which seems kind of crazy.
>
Yes you''re right, there are times when it''s better to bypass
the slog
and use the pool disks which can deliver better bandwidth.

The algorithm for where and what the ZIL writes has got quite complex:

- There was another change recently to bypass the slog if 1MB had been 
sent to it and 2MB were waiting to be sent.
- There''s a new property logbias which when set to throughput directs 
the ZIL to send all of it''s writes to the main pool devices thus
freeing
the slog for more latency sensitive work (ideal for database data files).
- If synchronous writes are large (>32K) and block aligned then the 
blocks are written directly to the pool and a small record
  written to the log. Later when the txg commits then the blocks are 
just linked into the txg. However, this processing is not
  done if there are any slogs because I found it didn''t perform as
well.
Probably ought to be re-evaluated.
- There are further tweaks being suggested and which might make it to a 
ZIL near you soon.

Neil.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100410/255fbf19/attachment.html>

On 04/10/10 14:55, Daniel Carosone wrote:
> On Sat, Apr 10, 2010 at 11:50:05AM -0500, Bob Friesenhahn wrote:
>   
>> Huge synchronous bulk writes are pretty rare since usually the 
>> bottleneck is elsewhere, such as the ethernet.
>>     
>
> Also, large writes can go straight to the pool, and the zil only logs
> the intent to commit those blocks (ie, link them into the zfs data
> structure).   I don''t recall what the threshold for this is, but I
> think it''s one of those Evil Tunables.
>   
This is zfs_immediate_write_sz which is 32K.  However this only happens
currently if you don''t have any slogs. If logbias is set to throughput
then
all writes go straight to the pool regardless of zfs_immediate_write_sz.

Neil.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100410/71965640/attachment.html>