Andrew Robb writes:
> The big problem that I have with non-directio is that buffering delays
> program execution. When reading/writing files that are many times
> larger than RAM without directio, it is very apparent that system
> response drops through the floor- it can take several minutes for an
> ssh login to prompt for a password. This is true both for UFS and
> ZFS.
Appart from the ZFS write, I find this a bit surprising. Are
you sure this is a general statement or would detail of your
configuration be of interest here. As for the ZFS write,
this problem is well on it''s way to being fixed.
6429205 each zpool needs to monitor it''s throughput and throttle heavy
writers
Now that you say this, I think I can see how it would be
possible in the UFS write case also. Both Read cases I find
troublesome though. See below on disable speculative reads.
For ZFS, I guess some relief might come from implementing something
like this :
6429855 Need way to tell ZFS that caching is a lost cause
>
> Repeat the exercise with directio on UFS and there is no discernible
> delay in starting new applications (ssh login takes a second or so to
> prompt for a password). Writing a large file might appear to take a
> few seconds longer with directio, but add a sync command to the end
> and it is apparent that there is no real delay in getting the data to
> disc with directio.
>
> I''d like to see directio() provide some of the facilities under
ZFS that it affords under UFS:
> 1. data is not buffered beyond the I/O request
> 2. no speculative reads
> 3. synchronous writes of whole records
> 4. concurrent I/O (which is already available in ZFS)
>
So I think we should not confuse UFS directio with
synchronous semantics. So I think point 3 comes from a
confusion.
For point 2, we''ve not too long ago fixed one level of
speculative reads (vdev) which should not cause problems
anymore. The other level (zfetch) needs attention. I see no
reason that good software can''t work out of the box for you.
In the mean time it is possible to disable speculative reads
as described here :
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#File-Level_Prefetching
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device-Level_Prefetching
For point 1, I think the current behavior of ZFS is far from
good. Once the problems are fixed though it will be time to
reevaluate if the buffering strategies are causing problems and
under what conditions. Your memory BW issues could be one of
them.
> Note: I consider memory bandwidth a finite and precious resource - not
> to be wasted in double-buffering. (I have a naive test program that is
> completely bound by main memory bandwidth - two programs on two CPUs
> run at half the speed of a single program on one CPU.)
>
But also consider you # disks and system bus bandwidth in
this equation. Many workloads won''t be hit by the double
memory BW if there are spindle limited. A lot of the longing
for Directio come from a few serious quirks in the current
implementation. You''ve had legitimate issues in UFS and ZFS
and the UFS issues happened to be fixed by UFS/DIO; I think many
of them can be fixed in ZFS with something not called Directio.
-r
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss