zfs discuss - Jul 2009 - When writing to SLOG at full speed all disk IO is blocked

Hello all...
 I''m seeing this behaviour in an old build (89), and i just want to
hear from you if there is some known bug about it. I''m aware of the
"picket fencing" problem, and that ZFS is not choosing right if write
to slog is better or not (thinking if we have a better throughput from disks).
 But i did not find anything about 100% slog activity (~115MB/s) blocks IO from
disks. Two or three seconds of "zero" read or write on disks...

 Thanks a lot for your time!

 Leal
[ http://www.eall.com.br/blog ]
-- 
This message posted from opensolaris.org

byleal,

Can you share how to recreate or test this?
-- 
This message posted from opensolaris.org

Hello,
 Well, i''m trying to understand this workload, but what i''m
seeing to reproduce this is just flood the SSD with writes, and the disks show
no activity. I''m testing with aggr (two links), and for one or two
seconds there is no read activity (output from server).
 Right now i''m suspecting something with the network, because i did
some ZFS tuning, and seems like i''m not getting the 100% utilization on
SSD, and the behaviour is still happening. I need to confirm this, and will
share with you.
 Thanks for your reply.

 Leal
[ http://www.eall.com.br/blog ]
-- 
This message posted from opensolaris.org

On Mon, 27 Jul 2009, Marcelo Leal wrote:
> Well, i''m trying to understand this workload, but what
i''m seeing to
> reproduce this is just flood the SSD with writes, and the disks show 
> no activity. I''m testing with aggr (two links), and for one or two
> seconds there is no read activity (output from server). Right now
In other situations we have noticed that writes take priority over 
reads in ZFS.  When ZFS writes a TXG, reads go away for a little 
while. In most server situations, we do want writes to take 
precedence.  If synchronous writes don''t take precedence then writers 
of important updates may be blocked due to many readers trying to 
access those important updates.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Ok Bob, but i think that is the problem about picket fencing... and so we are
talking about commit the sync operations to disk. What i''m seeing is no
read activity from disks when the slog is beeing written. The disks are
"zero" (no read, no write).

 Thanks a lot for your reply.

 Leal
[ http://www.eall.com.br/blog ]
-- 
This message posted from opensolaris.org

On Tue, 28 Jul 2009, Marcelo Leal wrote:
> Ok Bob, but i think that is the problem about picket fencing... and 
> so we are talking about commit the sync operations to disk. What
i''m
> seeing is no read activity from disks when the slog is beeing 
> written. The disks are "zero" (no read, no write).
This is an interesting issue.  While synchronous writes are requested, 
what do you expect a read to return?  If there is a synchronous write 
in progress, should readers wait for the write to be persisted in case 
the write influences the data read?

Note that I am not saying that huge synchronous writes should 
necessary block reading (particularly if the reads are for unrelated 
blocks/files), but it is understandable if zfs focuses more on the 
writes.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

My understanding is that there''s never any need for a reader to wait
for a write in progress.  ZFS keeps all writes in memory until they''re
committed to disk - if you ever try to read something that''s either
waiting to be, or is being written to disk, ZFS will serve it straight from RAM.

One question I do have after reading this again though is:

Leal, do you have the slog on the same controller as the disks?  Have you tested
whether reads are also blocked if you''re running on a separate
controller?
-- 
This message posted from opensolaris.org