Jonathan Chen
2016-Oct-20 22:48 UTC
zfs, a directory that used to hold lot of files and listing pause
On 21 October 2016 at 11:27, Steven Hartland <killing at multiplay.co.uk> wrote:> On 20/10/2016 22:18, Jonathan Chen wrote: >> >> On 21 October 2016 at 09:09, Peter <pmc at citylink.dinoex.sub.org> wrote: >> [...] >>> >>> I see this on my pgsql_tmp dirs (where Postgres stores intermediate >>> query data that gets too big for mem - usually lots of files) - in >>> normal operation these dirs are completely empty, but make heavy disk >>> activity (even writing!) when doing ls. >>> Seems normal, I dont care as long as the thing is stable. One would need >>> to check how ZFS stores directories and what kind of fragmentation can >>> happen there. Or wait for some future feature that would do >>> housekeeping. ;) >> >> I'm seeing this as well with an Odoo ERP running on Postgresql. This >> lag does matter to me as this is huge performance hit when running >> Postgresql on ZFS, and it would be good to see this resolved. >> pg_restores can make the system crawl as well. > > As mentioned before could you confirm you have disable atime?Yup, also set the blocksize to 4K. 11:46am# zfs get all irontree/postgresql NAME PROPERTY VALUE SOURCE irontree/postgresql type filesystem - irontree/postgresql creation Wed Sep 23 15:07 2015 - irontree/postgresql used 43.8G - irontree/postgresql available 592G - irontree/postgresql referenced 43.8G - irontree/postgresql compressratio 1.00x - irontree/postgresql mounted yes - irontree/postgresql quota none default irontree/postgresql reservation none default irontree/postgresql recordsize 8K local irontree/postgresql mountpoint /postgresql inherited from irontree irontree/postgresql sharenfs off default irontree/postgresql checksum on default irontree/postgresql compression off default irontree/postgresql atime off local irontree/postgresql devices on default irontree/postgresql exec on default irontree/postgresql setuid on default irontree/postgresql readonly off default irontree/postgresql jailed off default irontree/postgresql snapdir hidden default irontree/postgresql aclmode discard default irontree/postgresql aclinherit restricted default irontree/postgresql canmount on default irontree/postgresql xattr off temporary irontree/postgresql copies 1 default irontree/postgresql version 5 - irontree/postgresql utf8only off - irontree/postgresql normalization none - irontree/postgresql casesensitivity sensitive - irontree/postgresql vscan off default irontree/postgresql nbmand off default irontree/postgresql sharesmb off default irontree/postgresql refquota none default irontree/postgresql refreservation none default irontree/postgresql primarycache all default irontree/postgresql secondarycache all default irontree/postgresql usedbysnapshots 0 - irontree/postgresql usedbydataset 43.8G - irontree/postgresql usedbychildren 0 - irontree/postgresql usedbyrefreservation 0 - irontree/postgresql logbias latency default irontree/postgresql dedup off default irontree/postgresql mlslabel - irontree/postgresql sync standard default irontree/postgresql refcompressratio 1.00x - irontree/postgresql written 43.8G - irontree/postgresql logicalused 43.4G - irontree/postgresql logicalreferenced 43.4G - irontree/postgresql volmode default default irontree/postgresql filesystem_limit none default irontree/postgresql snapshot_limit none default irontree/postgresql filesystem_count none default irontree/postgresql snapshot_count none default irontree/postgresql redundant_metadata all default -- Jonathan Chen <jonc at chen.org.nz>
Steven Hartland
2016-Oct-20 23:56 UTC
zfs, a directory that used to hold lot of files and listing pause
On 20/10/2016 23:48, Jonathan Chen wrote:> On 21 October 2016 at 11:27, Steven Hartland <killing at multiplay.co.uk> wrote: >> On 20/10/2016 22:18, Jonathan Chen wrote: >>> On 21 October 2016 at 09:09, Peter <pmc at citylink.dinoex.sub.org> wrote: >>> [...] >>>> I see this on my pgsql_tmp dirs (where Postgres stores intermediate >>>> query data that gets too big for mem - usually lots of files) - in >>>> normal operation these dirs are completely empty, but make heavy disk >>>> activity (even writing!) when doing ls. >>>> Seems normal, I dont care as long as the thing is stable. One would need >>>> to check how ZFS stores directories and what kind of fragmentation can >>>> happen there. Or wait for some future feature that would do >>>> housekeeping. ;) >>> I'm seeing this as well with an Odoo ERP running on Postgresql. This >>> lag does matter to me as this is huge performance hit when running >>> Postgresql on ZFS, and it would be good to see this resolved. >>> pg_restores can make the system crawl as well. >> As mentioned before could you confirm you have disable atime? > Yup, also set the blocksize to 4K. > > 11:46am# zfs get all irontree/postgresql > NAME PROPERTY VALUE SOURCE > irontree/postgresql type filesystem - > irontree/postgresql creation Wed Sep 23 15:07 2015 - > irontree/postgresql used 43.8G - > irontree/postgresql available 592G - > irontree/postgresql referenced 43.8G - > irontree/postgresql compressratio 1.00x - > irontree/postgresql mounted yes - > irontree/postgresql quota none default > irontree/postgresql reservation none default > irontree/postgresql recordsize 8K local > irontree/postgresql mountpoint /postgresql > inherited from irontree > irontree/postgresql sharenfs off default > irontree/postgresql checksum on default > irontree/postgresql compression off default > irontree/postgresql atime off local > irontree/postgresql devices on default > irontree/postgresql exec on default > irontree/postgresql setuid on default > irontree/postgresql readonly off default > irontree/postgresql jailed off default > irontree/postgresql snapdir hidden default > irontree/postgresql aclmode discard default > irontree/postgresql aclinherit restricted default > irontree/postgresql canmount on default > irontree/postgresql xattr off temporary > irontree/postgresql copies 1 default > irontree/postgresql version 5 - > irontree/postgresql utf8only off - > irontree/postgresql normalization none - > irontree/postgresql casesensitivity sensitive - > irontree/postgresql vscan off default > irontree/postgresql nbmand off default > irontree/postgresql sharesmb off default > irontree/postgresql refquota none default > irontree/postgresql refreservation none default > irontree/postgresql primarycache all default > irontree/postgresql secondarycache all default > irontree/postgresql usedbysnapshots 0 - > irontree/postgresql usedbydataset 43.8G - > irontree/postgresql usedbychildren 0 - > irontree/postgresql usedbyrefreservation 0 - > irontree/postgresql logbias latency default > irontree/postgresql dedup off default > irontree/postgresql mlslabel - > irontree/postgresql sync standard default > irontree/postgresql refcompressratio 1.00x - > irontree/postgresql written 43.8G - > irontree/postgresql logicalused 43.4G - > irontree/postgresql logicalreferenced 43.4G - > irontree/postgresql volmode default default > irontree/postgresql filesystem_limit none default > irontree/postgresql snapshot_limit none default > irontree/postgresql filesystem_count none default > irontree/postgresql snapshot_count none default > irontree/postgresql redundant_metadata all default >When you see the stalling what does gstat -pd and top -SHz show? Regards Steve