zfs discuss - Nov 2009 - sched regularily writing a lots of MBs to the pool?

I''ve been noticing regular writing activity to my data pool while the
system''s relatively idle, just a little read IO. Turns out the
system''s writing up to 20MB of data to the pool every 15-30 seconds.
Using iotop from the DTrace Toolkit, apparently the process responsible is
sched. What''s going on here? Playing back a simple video file
wouldn''t generate enough metadata changes to create that much updates.

Thanks for any help,
-mg
-- 
This message posted from opensolaris.org

On Wed, 4 Nov 2009, Mario Goebbels wrote:
> I''ve been noticing regular writing activity to my data pool while 
> the system''s relatively idle, just a little read IO. Turns out the
> system''s writing up to 20MB of data to the pool every 15-30
seconds.
> Using iotop from the DTrace Toolkit, apparently the process 
> responsible is sched. What''s going on here? Playing back a simple 
> video file wouldn''t generate enough metadata changes to create
that
> much updates.
Did you disable ''atime'' updates for your filesystem? 
Otherwise the
file access times need to be periodically updated and this would 
happen may every 15-30 seconds.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

>> I''ve been noticing regular writing activity to my data pool
while the
>> system''s relatively idle, just a little read IO. Turns out the
>> system''s writing up to 20MB of data to the pool every 15-30
seconds.
>> Using iotop from the DTrace Toolkit, apparently the process
>> responsible is sched. What''s going on here? Playing back a
simple
>> video file wouldn''t generate enough metadata changes to create
that
>> much updates.
>
> Did you disable ''atime'' updates for your filesystem?
Otherwise the file
> access times need to be periodically updated and this would happen may
> every 15-30 seconds.
Not disabled. But 20MB worth of metadata updates while the system 
practically does nothing? Only real things happening is a video 
playbacking and XChat logging relatively few things to disk.

-mg

On Wed, 4 Nov 2009, Mario Goebbels wrote:
>> 
>> Did you disable ''atime'' updates for your filesystem?
Otherwise the file
>> access times need to be periodically updated and this would happen may
>> every 15-30 seconds.
>
> Not disabled. But 20MB worth of metadata updates while the system
practically
> does nothing? Only real things happening is a video playbacking and XChat 
> logging relatively few things to disk.
20MB would indeed be quite a lot.  On one of my systems,
''iotop''
periodically shows up to 8504 KB of writen data, and no reads.  This 
is for the root pool.  I have a pretty good idea of where that 8504 KB 
of data comes from, and it is due to a daemon I have running.  On 
another system (which still uses UFS root, but with most data in a ZFS 
pool), only a few tiny writes (biggest 810KB).

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

>>> Did you disable ''atime'' updates for your
filesystem? Otherwise the file
>>> access times need to be periodically updated and this would happen
may
>>> every 15-30 seconds.
>>
>> Not disabled. But 20MB worth of metadata updates while the system
>> practically does nothing? Only real things happening is a video
>> playbacking and XChat logging relatively few things to disk.
>
> 20MB would indeed be quite a lot. On one of my systems,
''iotop''
> periodically shows up to 8504 KB of writen data, and no reads. This is
> for the root pool. I have a pretty good idea of where that 8504 KB of
> data comes from, and it is due to a daemon I have running. On another
> system (which still uses UFS root, but with most data in a ZFS pool),
> only a few tiny writes (biggest 810KB).
I think I found a/the cause. Once Firefox was killed, it quasi stopped.

It''s probably related again to Firefox regularily rewriting all of its 
SQLite databases for no good reason.

Hrm.

-mg

zfs groups writes together into transaction groups; the physical writes
to disk are generally initiated by kernel threads (which appear in
dtrace as threads of the "sched" process).  Changing the attribution
is
not going to be simple as a single physical write to the pool may
contain data and metadata changes triggered by multiple user processes.

You need to go up a level of abstraction and look at the vnode layer to
attribute writes to particular processes.

> zfs groups writes together into transaction groups; the physical writes
> to disk are generally initiated by kernel threads (which appear in
> dtrace as threads of the "sched" process).  Changing the
attribution is
> not going to be simple as a single physical write to the pool may
> contain data and metadata changes triggered by multiple user processes.
>
> You need to go up a level of abstraction and look at the vnode layer to
> attribute writes to particular processes.
Vnode layer is fsinfo? I''ve tried one of the DTrace examples and get 
this while idle:

hyperborea:~ > pfexec dtrace -n
fsinfo:::write''{@[args[0]->fi_pathname]
= count()}''
dtrace: description ''fsinfo:::write'' matched 1 probe
^C

   /pacman/logs/xchat/NETWORK-#apb-200911.log                        2
 
/export/home/servo/.mozilla/firefox/mouyg8f7.default/sessionstore-2.js 
               4
   /export/home/servo/.mozilla/firefox/mouyg8f7.default/cookies.sqlite 
               5
 
/export/home/servo/.mozilla/firefox/mouyg8f7.default/cookies.sqlite-journal 
               17
   /export/home/servo/.xchat2/scrollback/Neowin/#neowin.txt               28
   /pacman/logs/xchat/Neowin-#neowin-200911.log                     28
   <unknown>                                                      3472


The lots of unknown writes don''t really help. Any other idea how I can 
track this down?

Thanks,
-mg