Steven Hartland
2016-Oct-20 23:56 UTC
zfs, a directory that used to hold lot of files and listing pause
On 20/10/2016 23:48, Jonathan Chen wrote:> On 21 October 2016 at 11:27, Steven Hartland <killing at multiplay.co.uk> wrote: >> On 20/10/2016 22:18, Jonathan Chen wrote: >>> On 21 October 2016 at 09:09, Peter <pmc at citylink.dinoex.sub.org> wrote: >>> [...] >>>> I see this on my pgsql_tmp dirs (where Postgres stores intermediate >>>> query data that gets too big for mem - usually lots of files) - in >>>> normal operation these dirs are completely empty, but make heavy disk >>>> activity (even writing!) when doing ls. >>>> Seems normal, I dont care as long as the thing is stable. One would need >>>> to check how ZFS stores directories and what kind of fragmentation can >>>> happen there. Or wait for some future feature that would do >>>> housekeeping. ;) >>> I'm seeing this as well with an Odoo ERP running on Postgresql. This >>> lag does matter to me as this is huge performance hit when running >>> Postgresql on ZFS, and it would be good to see this resolved. >>> pg_restores can make the system crawl as well. >> As mentioned before could you confirm you have disable atime? > Yup, also set the blocksize to 4K. > > 11:46am# zfs get all irontree/postgresql > NAME PROPERTY VALUE SOURCE > irontree/postgresql type filesystem - > irontree/postgresql creation Wed Sep 23 15:07 2015 - > irontree/postgresql used 43.8G - > irontree/postgresql available 592G - > irontree/postgresql referenced 43.8G - > irontree/postgresql compressratio 1.00x - > irontree/postgresql mounted yes - > irontree/postgresql quota none default > irontree/postgresql reservation none default > irontree/postgresql recordsize 8K local > irontree/postgresql mountpoint /postgresql > inherited from irontree > irontree/postgresql sharenfs off default > irontree/postgresql checksum on default > irontree/postgresql compression off default > irontree/postgresql atime off local > irontree/postgresql devices on default > irontree/postgresql exec on default > irontree/postgresql setuid on default > irontree/postgresql readonly off default > irontree/postgresql jailed off default > irontree/postgresql snapdir hidden default > irontree/postgresql aclmode discard default > irontree/postgresql aclinherit restricted default > irontree/postgresql canmount on default > irontree/postgresql xattr off temporary > irontree/postgresql copies 1 default > irontree/postgresql version 5 - > irontree/postgresql utf8only off - > irontree/postgresql normalization none - > irontree/postgresql casesensitivity sensitive - > irontree/postgresql vscan off default > irontree/postgresql nbmand off default > irontree/postgresql sharesmb off default > irontree/postgresql refquota none default > irontree/postgresql refreservation none default > irontree/postgresql primarycache all default > irontree/postgresql secondarycache all default > irontree/postgresql usedbysnapshots 0 - > irontree/postgresql usedbydataset 43.8G - > irontree/postgresql usedbychildren 0 - > irontree/postgresql usedbyrefreservation 0 - > irontree/postgresql logbias latency default > irontree/postgresql dedup off default > irontree/postgresql mlslabel - > irontree/postgresql sync standard default > irontree/postgresql refcompressratio 1.00x - > irontree/postgresql written 43.8G - > irontree/postgresql logicalused 43.4G - > irontree/postgresql logicalreferenced 43.4G - > irontree/postgresql volmode default default > irontree/postgresql filesystem_limit none default > irontree/postgresql snapshot_limit none default > irontree/postgresql filesystem_count none default > irontree/postgresql snapshot_count none default > irontree/postgresql redundant_metadata all default >When you see the stalling what does gstat -pd and top -SHz show? Regards Steve
Jonathan Chen
2016-Oct-21 00:54 UTC
zfs, a directory that used to hold lot of files and listing pause
On 21 October 2016 at 12:56, Steven Hartland <killing at multiplay.co.uk> wrote: [...]> When you see the stalling what does gstat -pd and top -SHz show?On my dev box: 1:38pm# uname -a FreeBSD irontree 10.3-STABLE FreeBSD 10.3-STABLE #0 r307401: Mon Oct 17 10:17:22 NZDT 2016 root at irontree:/usr/obj/usr/src/sys/GENERIC amd64 1:49pm# gstat -pd dT: 1.004s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| cd0 18 618 1 128 41.4 606 52854 17.2 0 0 0.0 100.5| ada0 ^C 1:49pm# top -SHz last pid: 83284; load averages: 0.89, 0.68, 0.46 up 4+03:11:32 13:49:05 565 processes: 9 running, 517 sleeping, 17 zombie, 22 waiting CPU: 3.7% user, 0.0% nice, 1.9% system, 0.0% interrupt, 94.3% idle Mem: 543M Active, 2153M Inact, 11G Wired, 10M Cache, 2132M Free ARC: 7249M Total, 1325M MFU, 4534M MRU, 906M Anon, 223M Header, 261M Other Swap: 32G Total, 201M Used, 32G Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 83149 postgres 38 0 2197M 528M zio->i 5 1:13 23.19% postgres 83148 jonc 22 0 36028K 13476K select 2 0:11 3.86% pg_restore 852 postgres 20 0 2181M 2051M select 5 0:27 0.68% postgres 0 root -15 - 0K 4240K - 6 0:50 0.49% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 6 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 6 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 7 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 7 0:50 0.39% kernel{zio_write_issue_} 0 root -15 - 0K 4240K - 7 0:50 0.29% kernel{zio_write_issue_} 3 root -8 - 0K 112K zio->i 6 1:50 0.20% zfskern{txg_thread_enter} 12 root -88 - 0K 352K WAIT 0 1:07 0.20% intr{irq268: ahci0} 0 root -16 - 0K 4240K - 4 0:29 0.20% kernel{zio_write_intr_4} 0 root -16 - 0K 4240K - 7 0:29 0.10% kernel{zio_write_intr_6} 0 root -16 - 0K 4240K - 0 0:29 0.10% kernel{zio_write_intr_1} 0 root -16 - 0K 4240K - 5 0:29 0.10% kernel{zio_write_intr_2} 0 root -16 - 0K 4240K - 1 0:29 0.10% kernel{zio_write_intr_5} ... Taking another look at the internal dir structure for postgres, I'm not too sure whether this is related to the original poster's problem though. Cheers. -- Jonathan Chen <jonc at chen.org.nz>