freebsd stable - Jul 2008 - msync() differences between Linux and FreeBSD

Hi,

I recently had a patch accepted to the ports collection that took out an msync()
call that seriously detrimented performance for rrdtool updates.  It seems that
in FreeBSD, msync() waits for bits to be committed to disk even when MS_ASYNC
is specified.  Under Linux, there is not such a wait for msync().

First off, I don't know how frequently msync() is used, and whether changing
its behavior would impact the performance of many things.

It's possible that FreeBSD msync() behavior is more correct in some ways.
From
a message from Matt Dillon on this same list:

    It used to be that msync() only synced VM pages to the underlying
    file, making them consistent with read()'s and write()'s against
    the underlying file.  Since FreeBSD uses a unified VM page cache
    this is always true.  However, the Open Group specification now
    requires that the dirty pages actually be written out to the underlying
    media... i.e. issue real I/O.  So msync() can't be a NOP if you go by
    the OpenGroup specification.

Is there a spec that FreeBSD is adhering to that prevents msync() with
MS_ASYNC from being a NOP, seeing as munmap() does the job?  And does this
really matter for the real-world performance of some apps?

Thanks,

Marcus

On 2008-Jul-02 05:00:02 -0700, Marcus Reid <marcus@blazingdot.com>
wrote:
>  It seems that in FreeBSD, msync() waits for bits to be
>committed to disk even when MS_ASYNC is specified.
Your previous ktrace output suggests that, at least for the way
rrdtool is using mmap(2), physical I/O is being performed by msync(2).
It's not clear whether FreeBSD is ignoring the MS_ASYNC flag (the code
suggests it isn't), is blocking on previously queued I/O or is blocking
for some other reason.
>First off, I don't know how frequently msync() is used, and whether
changing
>its behavior would impact the performance of many things.
The behaviour of msync(2) is defined in the Single Unix Specification
and FreeBSD adheres to SUS unless there is a very good reason.
>    media... i.e. issue real I/O.  So msync() can't be a NOP if you go
by
>    the OpenGroup specification.
>
>Is there a spec that FreeBSD is adhering to that prevents msync() with
>MS_ASYNC from being a NOP, seeing as munmap() does the job?
As per Matt's response that you quoted, yes.
>  And does this
>really matter for the real-world performance of some apps?
IMO, rrdtool is using mmap()/madvise()/msync()/munmap() in an unusual
fashion and it should be fixed, rather than changing FreeBSD to match
rrdtool.  I believe a more usual approach would be to mmap() a file
(or part thereof), optionally call madvise(), perform a series of
accesses and maybe msync()s of any updated regions then a single
munmap() before exiting.  Performing mmap()/msync()/munmap() (where
the msync() specifies the entire file) for each update maximises
system overheads for no obvious benefit.

Also, you mentioned hitting a "brick wall" between 940MB and 1161MB.
That straddles 1GB.  You may also be running into system or process
boundary conditions (how much RAM do you have and what tuning have you
done).  You might like to write a tool to simulate the rrdtool
behaviour with varying DB sizes and identify exactly what you are
hitting.

-- 
Peter Jeremy
Please excuse any delays as the result of my ISP's inability to implement
an MTA that is either RFC2821-compliant or matches their claimed behaviour.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080704/01d5aed9/attachment.pgp