Gluster users - Apr 2018 - performance.cache-size for high-RAM clients/servers, other tweaks for performance, and improvements to Gluster docs

Hi,

I'm trying to squeeze performance out of gluster on 4 80GB RAM 20-CPU
machines where Gluster runs on attached block storage (Linode) in (4
replicate bricks), and so far everything I tried results in sub-optimal
performance.

There are many files - mostly images, several million - and many operations
take minutes, copying multiple files (even if they're small) suddenly
freezes up for seconds at a time, then continues, iostat frequently shows
large r_await and w_awaits with 100% utilization for the attached block
device, etc.

But anyway, there are many guides out there for small-file performance
improvements, but more explanation is needed, and I think more tweaks
should be possible.

My question today is about performance.cache-size. Is this a size of cache
in RAM? If so, how do I view the current cache size to see if it gets full
and I should increase its size? Is it advisable to bump it up if I have
many tens of gigs of RAM free?



More generally, in the last 2 months since I first started working with
gluster and set a production system live, I've been feeling frustrated
because Gluster has a lot of poorly-documented and confusing options. I
really wish documentation could be improved with examples and better
explanations.

Specifically, it'd be absolutely amazing if the docs offered a strategy for
setting each value and ways of determining more optimal values. For
example, for performance.cache-size, if it said something like "run command
abc to see your current cache size, and if it's hurting, up it, but be
aware that it's limited by RAM," it'd be already a huge improvement
to the
docs. And so on with other options.



The gluster team is quite helpful on this mailing list, but in a reactive
rather than proactive way. Perhaps it's tunnel vision once you've worked
on
a project for so long where less technical explanations and even proper
documentation of options takes a back seat, but I encourage you to be more
proactive about helping us understand and optimize Gluster.

Thank you.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180405/37dfaa25/attachment.html>

Hi again,

I'd like to expand on the performance issues and plead for help. Here's
one
case which shows these odd hiccups: https://i.imgur.com/CXBPjTK.gifv.

In this GIF where I switch back and forth between copy operations on 2
servers, I'm copying a 10GB dir full of .apk and image files.

On server "hive" I'm copying straight from the main disk to an
attached
volume block (xfs). As you can see, the transfers are relatively speedy and
don't hiccup.
On server "citadel" I'm copying the same set of data to a
4-replicate
gluster which uses block storage as a brick. As you can see, performance is
much worse, and there are frequent pauses for many seconds where nothing
seems to be happening - just freezes.

All 4 servers have the same specs, and all of them have performance issues
with gluster and no such issues when raw xfs block storage is used.

hive has long finished copying the data, while citadel is barely chugging
along and is expected to take probably half an hour to an hour. I have over
1TB of data to migrate, at which point if we went live, I'm not even sure
gluster would be able to keep up instead of bringing the machines and
services down.



Here's the cluster config, though it didn't seem to make any difference
performance-wise before I applied the customizations vs after.

Volume Name: apkmirror_data1
Type: Replicate
Volume ID: 11ecee7e-d4f8-497a-9994-ceb144d6841e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: nexus2:/mnt/nexus2_block1/apkmirror_data1
Brick2: forge:/mnt/forge_block1/apkmirror_data1
Brick3: hive:/mnt/hive_block1/apkmirror_data1
Brick4: citadel:/mnt/citadel_block1/apkmirror_data1
Options Reconfigured:
cluster.quorum-count: 1
cluster.quorum-type: fixed
network.ping-timeout: 5
network.remote-dio: enable
performance.rda-cache-limit: 256MB
performance.readdir-ahead: on
performance.parallel-readdir: on
network.inode-lru-limit: 500000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
cluster.readdir-optimize: on
performance.io-thread-count: 32
server.event-threads: 4
client.event-threads: 4
performance.read-ahead: off
cluster.lookup-optimize: on
performance.cache-size: 1GB
cluster.self-heal-daemon: enable
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on


The mounts are done as follows in /etc/fstab:
/dev/disk/by-id/scsi-0Linode_Volume_citadel_block1 /mnt/citadel_block1 xfs
defaults 0 2
localhost:/apkmirror_data1 /mnt/apkmirror_data1 glusterfs defaults,_netdev
0 0

I'm really not sure if direct-io-mode mount tweaks would do anything here,
what the value should be set to, and what it is by default.

The OS is OpenSUSE 42.3, 64-bit. 80GB of RAM, 20 CPUs, hosted by Linode.

I'd really appreciate any help in the matter.

Thank you.


Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>

On Thu, Apr 5, 2018 at 11:13 PM, Artem Russakovskii <archon810 at
gmail.com>
wrote:
> Hi,
>
> I'm trying to squeeze performance out of gluster on 4 80GB RAM 20-CPU
> machines where Gluster runs on attached block storage (Linode) in (4
> replicate bricks), and so far everything I tried results in sub-optimal
> performance.
>
> There are many files - mostly images, several million - and many
> operations take minutes, copying multiple files (even if they're small)
> suddenly freezes up for seconds at a time, then continues, iostat
> frequently shows large r_await and w_awaits with 100% utilization for the
> attached block device, etc.
>
> But anyway, there are many guides out there for small-file performance
> improvements, but more explanation is needed, and I think more tweaks
> should be possible.
>
> My question today is about performance.cache-size. Is this a size of cache
> in RAM? If so, how do I view the current cache size to see if it gets full
> and I should increase its size? Is it advisable to bump it up if I have
> many tens of gigs of RAM free?
>
>
>
> More generally, in the last 2 months since I first started working with
> gluster and set a production system live, I've been feeling frustrated
> because Gluster has a lot of poorly-documented and confusing options. I
> really wish documentation could be improved with examples and better
> explanations.
>
> Specifically, it'd be absolutely amazing if the docs offered a strategy
> for setting each value and ways of determining more optimal values. For
> example, for performance.cache-size, if it said something like "run
command
> abc to see your current cache size, and if it's hurting, up it, but be
> aware that it's limited by RAM," it'd be already a huge
improvement to the
> docs. And so on with other options.
>
>
>
> The gluster team is quite helpful on this mailing list, but in a reactive
> rather than proactive way. Perhaps it's tunnel vision once you've
worked on
> a project for so long where less technical explanations and even proper
> documentation of options takes a back seat, but I encourage you to be more
> proactive about helping us understand and optimize Gluster.
>
> Thank you.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180406/8020a224/attachment.html>

I restarted rsync, and this has been sitting there for almost a minute,
barely moved several bytes in that time:

2014/11/545b06baa3d98/com.google.android.apps.inputmethod.zhuyin-2.1.0.79226761-armeabi-v7a-175-minAPI14.apk
      6,389,760  45%   18.76kB/s    0:06:50

I straced each of the 3 processes rsync created and saw this (note: every
time there were several seconds of no output, I ctrl-C'ed and detached from
strace):

citadel:/home/archon810 # strace -p 16776
Process 16776 attached
select(6, [5], [4], [5], {45, 293510})  = 1 (out [4], left {44, 71342})
write(4,
"\4\200\0\7\3513>\2755\360[\372\317\337DZ\36\324\300o\235\377\247\367\177%\37\226\352\377\256\351"...,
32776) = 32776
ioctl(1, TIOCGPGRP, [16776])            = 0
write(1, "\r      4,292,608  30%   27.07kB/"..., 46) = 46
select(6, [5], [4], [5], {60, 0})       = 1 (out [4], left {59, 999998})
write(4,
"\4\200\0\7\270\224\277\24\31\247f\32\233x\t\276l\f-\254\r\246\324\360\30\235\350\6\34\304\230\242"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})       = 1 (out [4], left {59, 999998})
write(4,
"\4\200\0\7\346_\363\36\33\320}Dd\5_\327\250\237i\242?B\276e\245\202Z\213\301[\25S"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})       = 1 (out [4], left {59, 999998})
write(4,
"\4\200\0\7\330\303\221\357\225\37h\373\366X\306L\f>\234\\%n\253\266\5\372c\257>V\366\255"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})       = 1 (out [4], left {59, 999998})
write(4,
"\4\200\0\7i\301\17u\224{/O\213\330\33\317\272\246\221\22\261|w\244\5\307|\21\373\v\356k"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})       = 1 (out [4], left {59, 999998})
write(4,
"\4\200\0\7\270\277\233\206n\304:\362_\213~\356bm\5\350\337\26\203\225\332\277\372\275\247<\307\22"...,
32776) = 32776
read(3, "\316\214\260\341:\263P\214\373n\313\10\333
}\323\364Q\353\r\232d\204\257\\Q\306/\277\253/\356"..., 262144) = 262144
select(6, [5], [4], [5], {60, 0})       = 1 (out [4], left {59, 999998})
write(4,
"\4\200\0\7\314\233\274S08\330\276\226\267\233\360rp\210x)\320\0314\223\323\3335Y\312\313\307"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})       = 1 (out [4], left {59, 999998})
write(4, "\4\200\0\7\316\214\260\341:\263P\214\373n\313\10\333
}\323\364Q\353\r\232d\204\257\\Q\306/"..., 32776) = 32776
select(6, [5], [4], [5], {60, 0}^CProcess 16776 detached
 <detached ...>
citadel:/home/archon810 # strace -p 16777
Process 16777 attached
select(4, [3], [], [3], {38, 210908}^CProcess 16777 detached
 <detached ...>
citadel:/home/archon810 # strace -p 16776
Process 16776 attached
select(6, [5], [4], [5], {48, 295874}^CProcess 16776 detached
 <detached ...>
citadel:/home/archon810 # strace -p 16778
Process 16778 attached
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999996})
read(0,
"\0\200\0\0\4\200\0\7\3508\343\204\207\255\4\212y\230&&\372\30*\322\f\325v\335\230
\16v"..., 32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999998})
read(0, "\373\30\2\2667\371\207)", 8)   = 8
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0,
"\0\200\0\0\4\200\0\7\6\213\2223\233\36-\350,\303\0\234\7`\317\276H\353u\217\275\316\333@"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0, "\375\33\367_\357\330\362\222", 8) = 8
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0,
"\0\200\0\0\4\200\0\7`Nv\355\275\336wzQ\365\264\364\20AX\365DG\372\311\216\212\375\276"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0, "\374)\300\264}\21\226s", 8)    = 8
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0, "\0\200\0\0\4\200\0\7\10:\v\342O\305\374\5:Y+
\250\315\24\202J-@\256WC\320\371"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0, "\3023\24O\343y\312\204", 8)    = 8
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0,
"\0\200\0\0\4\200\0\7\27\22^\n/S.\215\362T\f\257Q\207z\241~B\3\32\32344\17"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999998})
read(0, "\367P\222\262\224\17\25\250", 8) = 8
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0,
"\0\200\0\0\4\200\0\7FujR\213\372\341E\232\360\n\257\323\233>\364\245\37\3\31\314\20\206\362"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0, "\203o\300\341\37\340(8", 8)    = 8
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999998})
read(0,
"\0\200\0\0\4\200\0\7n\211\357\301\217\210\23\341$\342d8\25N\2035[\260\1\206B\206!\2"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0, "|\222\223\336\201w\325\356", 8) = 8
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999999})
read(0,
"\0\200\0\0\4\200\0\7\220\216Y\343\362\366\231\372?\334N^\303\35\374cC;\vtx\231<w"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})        = 1 (in [0], left {59, 999998})
read(0, ";k\v\314\21\375\3\274", 8)     = 8
write(3,
"\3508\343\204\207\255\4\212y\230&&\372\30*\322\f\325v\335\230
\16v\213O//\332\4\24\24"..., 262144^C

I'm really not sure what to make of this. In the time I wrote the above,
the file still hasn't finished copying.

2014/11/545b06baa3d98/com.google.android.apps.inputmethod.zhuyin-2.1.0.79226761-armeabi-v7a-175-minAPI14.apk
     10,321,920  73%   33.31kB/s    0:01:53


Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>

On Fri, Apr 6, 2018 at 1:12 AM, Artem Russakovskii <archon810 at
gmail.com>
wrote:
> Hi again,
>
> I'd like to expand on the performance issues and plead for help.
Here's
> one case which shows these odd hiccups: https://i.imgur.com/CXBPjTK.gifv.
>
> In this GIF where I switch back and forth between copy operations on 2
> servers, I'm copying a 10GB dir full of .apk and image files.
>
> On server "hive" I'm copying straight from the main disk to
an attached
> volume block (xfs). As you can see, the transfers are relatively speedy and
> don't hiccup.
> On server "citadel" I'm copying the same set of data to a
4-replicate
> gluster which uses block storage as a brick. As you can see, performance is
> much worse, and there are frequent pauses for many seconds where nothing
> seems to be happening - just freezes.
>
> All 4 servers have the same specs, and all of them have performance issues
> with gluster and no such issues when raw xfs block storage is used.
>
> hive has long finished copying the data, while citadel is barely chugging
> along and is expected to take probably half an hour to an hour. I have over
> 1TB of data to migrate, at which point if we went live, I'm not even
sure
> gluster would be able to keep up instead of bringing the machines and
> services down.
>
>
>
> Here's the cluster config, though it didn't seem to make any
difference
> performance-wise before I applied the customizations vs after.
>
> Volume Name: apkmirror_data1
> Type: Replicate
> Volume ID: 11ecee7e-d4f8-497a-9994-ceb144d6841e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 4 = 4
> Transport-type: tcp
> Bricks:
> Brick1: nexus2:/mnt/nexus2_block1/apkmirror_data1
> Brick2: forge:/mnt/forge_block1/apkmirror_data1
> Brick3: hive:/mnt/hive_block1/apkmirror_data1
> Brick4: citadel:/mnt/citadel_block1/apkmirror_data1
> Options Reconfigured:
> cluster.quorum-count: 1
> cluster.quorum-type: fixed
> network.ping-timeout: 5
> network.remote-dio: enable
> performance.rda-cache-limit: 256MB
> performance.readdir-ahead: on
> performance.parallel-readdir: on
> network.inode-lru-limit: 500000
> performance.md-cache-timeout: 600
> performance.cache-invalidation: on
> performance.stat-prefetch: on
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: on
> cluster.readdir-optimize: on
> performance.io-thread-count: 32
> server.event-threads: 4
> client.event-threads: 4
> performance.read-ahead: off
> cluster.lookup-optimize: on
> performance.cache-size: 1GB
> cluster.self-heal-daemon: enable
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: on
>
>
> The mounts are done as follows in /etc/fstab:
> /dev/disk/by-id/scsi-0Linode_Volume_citadel_block1 /mnt/citadel_block1
> xfs defaults 0 2
> localhost:/apkmirror_data1 /mnt/apkmirror_data1 glusterfs defaults,_netdev
> 0 0
>
> I'm really not sure if direct-io-mode mount tweaks would do anything
here,
> what the value should be set to, and what it is by default.
>
> The OS is OpenSUSE 42.3, 64-bit. 80GB of RAM, 20 CPUs, hosted by Linode.
>
> I'd really appreciate any help in the matter.
>
> Thank you.
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
> On Thu, Apr 5, 2018 at 11:13 PM, Artem Russakovskii <archon810 at
gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to squeeze performance out of gluster on 4 80GB RAM
20-CPU
>> machines where Gluster runs on attached block storage (Linode) in (4
>> replicate bricks), and so far everything I tried results in sub-optimal
>> performance.
>>
>> There are many files - mostly images, several million - and many
>> operations take minutes, copying multiple files (even if they're
small)
>> suddenly freezes up for seconds at a time, then continues, iostat
>> frequently shows large r_await and w_awaits with 100% utilization for
the
>> attached block device, etc.
>>
>> But anyway, there are many guides out there for small-file performance
>> improvements, but more explanation is needed, and I think more tweaks
>> should be possible.
>>
>> My question today is about performance.cache-size. Is this a size of
>> cache in RAM? If so, how do I view the current cache size to see if it
gets
>> full and I should increase its size? Is it advisable to bump it up if I
>> have many tens of gigs of RAM free?
>>
>>
>>
>> More generally, in the last 2 months since I first started working with
>> gluster and set a production system live, I've been feeling
frustrated
>> because Gluster has a lot of poorly-documented and confusing options. I
>> really wish documentation could be improved with examples and better
>> explanations.
>>
>> Specifically, it'd be absolutely amazing if the docs offered a
strategy
>> for setting each value and ways of determining more optimal values. For
>> example, for performance.cache-size, if it said something like
"run command
>> abc to see your current cache size, and if it's hurting, up it, but
be
>> aware that it's limited by RAM," it'd be already a huge
improvement to the
>> docs. And so on with other options.
>>
>>
>>
>> The gluster team is quite helpful on this mailing list, but in a
reactive
>> rather than proactive way. Perhaps it's tunnel vision once
you've worked on
>> a project for so long where less technical explanations and even proper
>> documentation of options takes a back seat, but I encourage you to be
more
>> proactive about helping us understand and optimize Gluster.
>>
>> Thank you.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK
Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180406/150c6751/attachment.html>

you definitely need mount options to /etc/fstab
use ones from here
http://lists.gluster.org/pipermail/gluster-users/2018-April/033811.html

I went on with using local mounts to achieve performance as well

Also, 3.12 or 3.10 branches would be preferable for production

On Fri, Apr 6, 2018 at 4:12 AM, Artem Russakovskii <archon810 at
gmail.com>
wrote:
> Hi again,
>
> I'd like to expand on the performance issues and plead for help.
Here's
> one case which shows these odd hiccups: https://i.imgur.com/CXBPjTK.gifv.
>
> In this GIF where I switch back and forth between copy operations on 2
> servers, I'm copying a 10GB dir full of .apk and image files.
>
> On server "hive" I'm copying straight from the main disk to
an attached
> volume block (xfs). As you can see, the transfers are relatively speedy and
> don't hiccup.
> On server "citadel" I'm copying the same set of data to a
4-replicate
> gluster which uses block storage as a brick. As you can see, performance is
> much worse, and there are frequent pauses for many seconds where nothing
> seems to be happening - just freezes.
>
> All 4 servers have the same specs, and all of them have performance issues
> with gluster and no such issues when raw xfs block storage is used.
>
> hive has long finished copying the data, while citadel is barely chugging
> along and is expected to take probably half an hour to an hour. I have over
> 1TB of data to migrate, at which point if we went live, I'm not even
sure
> gluster would be able to keep up instead of bringing the machines and
> services down.
>
>
>
> Here's the cluster config, though it didn't seem to make any
difference
> performance-wise before I applied the customizations vs after.
>
> Volume Name: apkmirror_data1
> Type: Replicate
> Volume ID: 11ecee7e-d4f8-497a-9994-ceb144d6841e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 4 = 4
> Transport-type: tcp
> Bricks:
> Brick1: nexus2:/mnt/nexus2_block1/apkmirror_data1
> Brick2: forge:/mnt/forge_block1/apkmirror_data1
> Brick3: hive:/mnt/hive_block1/apkmirror_data1
> Brick4: citadel:/mnt/citadel_block1/apkmirror_data1
> Options Reconfigured:
> cluster.quorum-count: 1
> cluster.quorum-type: fixed
> network.ping-timeout: 5
> network.remote-dio: enable
> performance.rda-cache-limit: 256MB
> performance.readdir-ahead: on
> performance.parallel-readdir: on
> network.inode-lru-limit: 500000
> performance.md-cache-timeout: 600
> performance.cache-invalidation: on
> performance.stat-prefetch: on
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: on
> cluster.readdir-optimize: on
> performance.io-thread-count: 32
> server.event-threads: 4
> client.event-threads: 4
> performance.read-ahead: off
> cluster.lookup-optimize: on
> performance.cache-size: 1GB
> cluster.self-heal-daemon: enable
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: on
>
>
> The mounts are done as follows in /etc/fstab:
> /dev/disk/by-id/scsi-0Linode_Volume_citadel_block1 /mnt/citadel_block1
> xfs defaults 0 2
> localhost:/apkmirror_data1 /mnt/apkmirror_data1 glusterfs defaults,_netdev
> 0 0
>
> I'm really not sure if direct-io-mode mount tweaks would do anything
here,
> what the value should be set to, and what it is by default.
>
> The OS is OpenSUSE 42.3, 64-bit. 80GB of RAM, 20 CPUs, hosted by Linode.
>
> I'd really appreciate any help in the matter.
>
> Thank you.
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
> On Thu, Apr 5, 2018 at 11:13 PM, Artem Russakovskii <archon810 at
gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to squeeze performance out of gluster on 4 80GB RAM
20-CPU
>> machines where Gluster runs on attached block storage (Linode) in (4
>> replicate bricks), and so far everything I tried results in sub-optimal
>> performance.
>>
>> There are many files - mostly images, several million - and many
>> operations take minutes, copying multiple files (even if they're
small)
>> suddenly freezes up for seconds at a time, then continues, iostat
>> frequently shows large r_await and w_awaits with 100% utilization for
the
>> attached block device, etc.
>>
>> But anyway, there are many guides out there for small-file performance
>> improvements, but more explanation is needed, and I think more tweaks
>> should be possible.
>>
>> My question today is about performance.cache-size. Is this a size of
>> cache in RAM? If so, how do I view the current cache size to see if it
gets
>> full and I should increase its size? Is it advisable to bump it up if I
>> have many tens of gigs of RAM free?
>>
>>
>>
>> More generally, in the last 2 months since I first started working with
>> gluster and set a production system live, I've been feeling
frustrated
>> because Gluster has a lot of poorly-documented and confusing options. I
>> really wish documentation could be improved with examples and better
>> explanations.
>>
>> Specifically, it'd be absolutely amazing if the docs offered a
strategy
>> for setting each value and ways of determining more optimal values. For
>> example, for performance.cache-size, if it said something like
"run command
>> abc to see your current cache size, and if it's hurting, up it, but
be
>> aware that it's limited by RAM," it'd be already a huge
improvement to the
>> docs. And so on with other options.
>>
>>
>>
>> The gluster team is quite helpful on this mailing list, but in a
reactive
>> rather than proactive way. Perhaps it's tunnel vision once
you've worked on
>> a project for so long where less technical explanations and even proper
>> documentation of options takes a back seat, but I encourage you to be
more
>> proactive about helping us understand and optimize Gluster.
>>
>> Thank you.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK
Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180410/978d2dc1/attachment.html>