Gluster users - Apr 2018 - cluster.readdir-optimize and disappearing files/dirs bug

Hi,

As many of you know, gluster suffers from pretty bad performance issues
when there are lots of files. One way to at least attempt to improve
performance is setting cluster.readdir-optimize to on.

However, based on my recent tests (using Gluster 3.13.2), as well as tests
of many others (like
http://lists.gluster.org/pipermail/gluster-devel/2016-November/051417.html),
there's a bug that frequently makes the listings of entire dirs disappear.
Though the data is still there, it's a very scary bug for production
systems which may think all data is gone.

I know Gluster 4 was recently released, but it makes no mention
of cluster.readdir-optimize in the notes.

What's the status of diagnosing this bug and fixing it? We're
experiencing
major performance issues with gluster in production, and I'd love to do
everything possible to get them resolved.

Thank you.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180403/65ea3b1b/attachment.html>

I was able to make it out of that issue with listings by recreating
everything on 3.12.X
Or maybe it was some setting conflict as I ran tests of each setting for
performance and left only those that gave better results. End up with
settings below for the distributed mail-server using maildir
cluster.readdir-optimize didn't do much anyway

fuse mounting with
negative-timeout=10,attribute-timeout=30,fopen-keep-cache,direct-io-mode=enable,fetch-attempts=5

server.allow-insecure: on
cluster.shd-max-threads: 12
performance.rda-cache-limit: 128MB
cluster.readdir-optimize: on
cluster.read-hash-mode: 0
performance.strict-o-direct: on
cluster.lookup-unhashed: auto
performance.nl-cache: on
performance.nl-cache-timeout: 600
cluster.lookup-optimize: on
client.event-threads: 4
performance.client-io-threads: on
performance.md-cache-timeout: 600
server.event-threads: 4
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
network.inode-lru-limit: 90000
performance.cache-refresh-timeout: 10
performance.enable-least-priority: off
performance.cache-size: 2GB
cluster.nufa: on
cluster.choose-local: on
server.outstanding-rpc-limit: 128
disperse.eager-lock: off
performance.io-thread-count: 8


On Tue, Apr 3, 2018 at 1:43 PM, Artem Russakovskii <archon810 at
gmail.com>
wrote:
> Hi,
>
> As many of you know, gluster suffers from pretty bad performance issues
> when there are lots of files. One way to at least attempt to improve
> performance is setting cluster.readdir-optimize to on.
>
> However, based on my recent tests (using Gluster 3.13.2), as well as tests
> of many others (like http://lists.gluster.org/
> pipermail/gluster-devel/2016-November/051417.html), there's a bug that
> frequently makes the listings of entire dirs disappear. Though the data is
> still there, it's a very scary bug for production systems which may
think
> all data is gone.
>
> I know Gluster 4 was recently released, but it makes no mention
> of cluster.readdir-optimize in the notes.
>
> What's the status of diagnosing this bug and fixing it? We're
experiencing
> major performance issues with gluster in production, and I'd love to do
> everything possible to get them resolved.
>
> Thank you.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180403/50ca5a5b/attachment.html>

Can you check whether you are hitting
https://bugzilla.redhat.com/show_bug.cgi?id=1512437? Note that the fix is
not backported to 3.13 branch, but is available on 4.0 through
https://bugzilla.redhat.com/1512437.

On Tue, Apr 3, 2018 at 11:13 PM, Artem Russakovskii <archon810 at
gmail.com>
wrote:
> Hi,
>
> As many of you know, gluster suffers from pretty bad performance issues
> when there are lots of files. One way to at least attempt to improve
> performance is setting cluster.readdir-optimize to on.
>
> However, based on my recent tests (using Gluster 3.13.2), as well as tests
> of many others (like http://lists.gluster.org/
> pipermail/gluster-devel/2016-November/051417.html), there's a bug that
> frequently makes the listings of entire dirs disappear. Though the data is
> still there, it's a very scary bug for production systems which may
think
> all data is gone.
>
> I know Gluster 4 was recently released, but it makes no mention
> of cluster.readdir-optimize in the notes.
>
> What's the status of diagnosing this bug and fixing it? We're
experiencing
> major performance issues with gluster in production, and I'd love to do
> everything possible to get them resolved.
>
> Thank you.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180405/1b0dcffb/attachment.html>

After updating the test server to 4.0.1, I can indeed confirm that so far
the disappearing directories bug is gone.

On Wed, Apr 4, 2018, 8:29 PM Raghavendra Gowdappa <rgowdapp at redhat.com>
wrote:
> Can you check whether you are hitting
> https://bugzilla.redhat.com/show_bug.cgi?id=1512437? Note that the fix is
> not backported to 3.13 branch, but is available on 4.0 through
> https://bugzilla.redhat.com/1512437.
>
> On Tue, Apr 3, 2018 at 11:13 PM, Artem Russakovskii <archon810 at
gmail.com>
> wrote:
>
>> Hi,
>>
>> As many of you know, gluster suffers from pretty bad performance issues
>> when there are lots of files. One way to at least attempt to improve
>> performance is setting cluster.readdir-optimize to on.
>>
>> However, based on my recent tests (using Gluster 3.13.2), as well as
>> tests of many others (like
>>
http://lists.gluster.org/pipermail/gluster-devel/2016-November/051417.html),
>> there's a bug that frequently makes the listings of entire dirs
disappear.
>> Though the data is still there, it's a very scary bug for
production
>> systems which may think all data is gone.
>>
>> I know Gluster 4 was recently released, but it makes no mention
>> of cluster.readdir-optimize in the notes.
>>
>> What's the status of diagnosing this bug and fixing it? We're
>> experiencing major performance issues with gluster in production, and
I'd
>> love to do everything possible to get them resolved.
>>
>> Thank you.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK
Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180405/8245612a/attachment.html>