freebsd stable - Mar 2015 - RELENG_10 performance regression (was Re: 35-40% performance drop releng9 vs releng10 openvpn

On Fri, Mar 20, 2015 at 04:53:42PM -0700, Xin Li wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> On 03/20/15 14:02, Mike Tancsa wrote:
> > OK, I think I found where the RELENG_10 performance loss happened.
> > It seems 
> >
https://lists.freebsd.org/pipermail/svn-src-stable-10/2015-March/004778.html
> >
> >  is the issue.
> > 
> > Testing with a kernel from r279796 I get 76-77Mb of throughput.
> > With r279848 it drops to about 60Mb
> 
> Hrm, looking at 'diffgraph.svg', it suggest that writing through
devfs
> have incurred a tax on writev(2), which is basically this codepath, if
> my reading is correct, in sys/fs/devfs/devfs.c:
> 
>    1676 static int
>    1677 devfs_write_f(struct file *fp, struct uio *uio, struct ucred
> *cred,
>    1678     int flags, struct thread *td)
>    1679 {
> ...
>    1703         if (uio->uio_resid != resid || (error == 0 &&
resid !> 0)) {
>    1704                 vfs_timestamp(&dev->si_ctime);
>    1705                 dev->si_mtime = dev->si_ctime;
>    1706         }
> 
> Further looking at the code, in devfs vfs_timestamp() is also called
> when reading from a device node.  When the setting was 0, the code
> would return time_second directly instead of attempting to read the
> timestamp.
> 
> For the purpose of devfs, does it make sense to bump timestamps like
> normal filesystems for each read/write operation?  Looks like Mac OS X
> will bump timestamps for each operation but Debian don't.
First question is, what timecounter hardware is used.  I would accept
some slowdown from hardware like HPET, but it is indeed surprising
if caused by TSC.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 03/20/15 17:15, Konstantin Belousov wrote:
>> For the purpose of devfs, does it make sense to bump timestamps
>> like normal filesystems for each read/write operation?  Looks
>> like Mac OS X will bump timestamps for each operation but Debian
>> don't.
> 
> First question is, what timecounter hardware is used.  I would
> accept some slowdown from hardware like HPET, but it is indeed
> surprising if caused by TSC.
It was HPET (see earlier discussion where a FlameGraph is posted).

I've done a survey on a few other operating systems, including
OpenIndiana, Debian and OS X.  Neither OpenIndiana nor Debian update
timestamp for read/write events so I'd propose this change to make it
a runtime configurable behavior and disable by default:

	https://reviews.freebsd.org/D2104

Cheers,
- -- 
Xin LI <delphij at delphij.net>    https://www.delphij.net/
FreeBSD - The Power to Serve!           Live free or die
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.2 (FreeBSD)

iQIcBAEBCgAGBQJVDLscAAoJEJW2GBstM+nsGtMQAKyh3kW6VWPnCd353KELqeoE
3n295iHqVjAF6vszt71cfiAqlz0oSH+5i2SfR1LKVXH4cb2gMbXYbMeJVQiV7L93
8xEsBNtPF4Z/fxwyEBfFb3WcXHo+TGjvaCEbsuJIFKhqa4GPY230/7JELN/LXuNd
HJlP7YECNPgeVWyYuCdZkLu/BMOPjL44/wcC/YuXSQP65TK54kx59KLg671o9Z2R
ffpljAmJSqpNjas41qyVN+SNxr9ga8oQA0LTf8lbn3r4o2mx+te8yzNcyIn/gCU8
jawDGfGNiEvCTr76ST6rvELy4jFbmZyNFrbu+TA6VDT2/CCh1x5qhDceXTnZKPtX
iJFpDKsl87R5aXdRQjwCZKkTdaJ7AuRAG0lrmumxIlblgDmaDsVbmaqR1CU0JzWI
rvGblL6iwm/4pL+O9Igkr6oN6+yd1/9KM2yCB5BZDokLnoj6INWOft4z6jGyOUWh
T3oNVfB48g/j5NKRyU/vFvJIUix2mbE5ziem+RyLU6x/UOdYQJsKRIlN6xPiIx73
/G7kpT1JbsoS47fyere5jeUV2UOurRuH5UfBHjJCJ0X44tzrsMhTm3Vs2cOzJP7K
6SNbyxyCbX3uJN1Aq9nhg2scIVieV1/BRLoeI6LEVVZKTTig4LA8ekzb0P6e1Fwg
f9rvXNwpdeZPmsDwdpff
=Uxnt
-----END PGP SIGNATURE-----

On 3/20/2015 8:15 PM, Konstantin Belousov wrote:
>>
>> For the purpose of devfs, does it make sense to bump timestamps like
>> normal filesystems for each read/write operation?  Looks like Mac OS X
>> will bump timestamps for each operation but Debian don't.
>
> First question is, what timecounter hardware is used.  I would accept
> some slowdown from hardware like HPET, but it is indeed surprising
> if caused by TSC.
>
>
David Wolfskill suggested trying the problem commit with

vfs.timestamp_precision=0

and it does indeed restore performance to what it was.  The raw dtrace 
files are available and FlameGraphs can all be found at

http://tancsa.com/time/

	---Mike
-- 
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/